First draft Tech stack file provider by tinvaan · Pull Request #12 · sourcegraph/openctx

tinvaan · 2024-01-05T21:48:30Z

A first draft implementation of a tech stack file opencodegraph provider. Currently only supports npm packages and .js & .ts source files.

sqs · 2024-01-06T07:39:27Z

Awesome! I'll take a look this weekend. Any screenshots to share, or need help getting it actually running end-to-end?

tinvaan · 2024-01-06T11:36:54Z

Awesome! I'll take a look this weekend. Any screenshots to share, or need help getting it actually running end-to-end?

Yeah, I'm trying to get the annotations show up on the playground. It seems alright in a unit test.

tinvaan · 2024-01-07T11:24:36Z

Awesome! I'll take a look this weekend. Any screenshots to share, or need help getting it actually running end-to-end?

Yeah, I'm trying to get the annotations show up on the playground. It seems alright in a unit test.

The issue seems to be with loading server side modules(viz. fs, path & yaml) on the browser. For this provider, we need a way to read the settings defined in a techstack.yml file.

failed to get annotations: Error: Module "path" has been externalized for browser compatibility.
Cannot access "path.resolve" in client code. 
See https://vitejs.dev/guide/troubleshooting.html#module-externalized-for-browser-compatibility for more details.

I read up about polyfilling node libraries in vite.config as a workaround but I'm not too familiar with the frontend stack (vite etc), any suggestions on a workaround @sqs ?

tinvaan · 2024-01-07T21:33:17Z

Okay, got a POC for the provider up and running on the web playground - I'll polish the PR shortly.

sqs

Thank you! Some comments and questions. This is super helpful for designing the OpenCodeGraph provider API (which is super early).

(BTW, I have a big refactor of the OpenCodeGraph API in the provider-docs branch. It should be a very small modification to your provider code here, so that need not block merging this.)

client/web-playground/src/demo/settings.ts

sqs · 2024-01-08T09:15:49Z

provider/techstack/index.test.ts

+        const result = techstack.capabilities({}, SETTINGS)
+
+        expect(result).toBeDefined()
+        expect(result).toStrictEqual<CapabilitiesResult>({


I think we'd want the techstack file to show up not in code files but in config and build files, so maybe something like:

.* (dotfiles)

**/package.json

**/*.bazel

**/README.md

techstack.y?(a)ml of course

others like that?

Oh, or is the intent to show it alongside the require or import statements in JS/TS files? Can you add a README describing the expected behavior briefly?

Yes, that was the intention for the first draft implementation(only imports), it would be too complex and noisy to annotate rest of the code.

Annotating build and config files also makes sense, but I'd like to introduce that in a future iteration. I've intentionally kept this first draft minimal. Does this approach sound okay?

provider/techstack/index.ts

sqs · 2024-01-08T09:21:20Z

provider/techstack/index.ts

+ * @param filename - techstack yml filename
+ * @returns parsed yaml object
+ */
+async function load(filename: string): Promise<TSF> {


Being able to automatically infer the techstack.yml file seems quite important here. Unfortunately, OpenCodeGraph does not yet have a way to help you do this. I was thinking that providers could say in their capabilities response something like "please also send me the contents of the following files with each annotations request: techstack.yml, ...". What do you think about that approach?

I'm not sure I entirely follow you here but here are my thoughts.

I didn't intend to automatically infer the techstack.yml file - it's set explicitly in the provider settings and the load() method simply references that in L59 and I find the explicitness in this design better.

const report = await load(settings.yaml)

That said, it's maybe not impossible to automatically infer a techstack.yml file for a respository and supplement it with an appropriate override in provider settings. You could also introduce an optional member in the AnnotationsParams interface that sends any techstack.yml data for each annotation request.

However, I don't see any real gain in these approaches vs the current settings driven design. Just my 2 cents.

Fyi, I've dropped all path resolutions and I'm now adhering to reading techstack.yml content from an absolute path provided in the provider settings. I'll document this in a README, see the below commits.

25c128d

6757344

tinvaan · 2024-01-09T08:34:46Z

client/vscode/test/fixtures/workspace/.vscode/settings.json

    // "https://sourcegraph.test:3443/.api/opencodegraph": true,
    // "../../../../../../provider/hello-world/dist/index.js": true,
    "https://opencodegraph.org/npm/@opencodegraph/provider-hello-world": true,
+    "https://opencodegraph.org/npm/@opencodegraph/provider-techstack": {


@sqs can you help me host this provider at opencodegraph.org? how can I test the provider on VSCode locally?

opencodegraph.org/npm/... proxies any npm module. So it should already work!

sqs · 2024-01-22T19:21:52Z

Thank you @tinvaan. I will follow up on this when I finish a big migration in the API. You are the first to contribute something, which is awesome, but it means you will definitely feel the rapid change during this experimental period, sorry!

tinvaan · 2024-01-23T03:29:55Z

Thank you @tinvaan. I will follow up on this when I finish a big migration in the API. You are the first to contribute something, which is awesome, but it means you will definitely feel the rapid change during this experimental period, sorry!

That's fine @sqs , thanks for the heads up re the migration. Before your changes land, could you tag & release the current main branch? I'm planning to implement another provider and would be easier if I have a locked down release version that I can work with.

Lines not matching import regex are needed to preserve line numbers.

s/OpenCodeGraph/OpenCtx

sqs · 2024-05-28T01:22:54Z

➞  pnpm run test:unit                                                                                                                                                                              
 WARN  Unsupported engine: wanted: {"node":">=20.10.0"} (current: {"node":"v18.13.0","pnpm":"8.14.1"})

> @openctx/root@ test:unit /Users/harish/Workspaces/oss/sourcegraph/opencodegraph
> vitest run

Error: Build failed with 1 error:
renderer/+config.ts:3:20: ERROR: [plugin: vike-esbuild] Could not resolve "/logomark-v0.svg"
 ELIFECYCLE  Command failed with exit code 1.

I think this is from an old version of the web/ code. I recommend rebasing your PR on top of latest main.

sqs requested changes Jan 8, 2024

View reviewed changes

tinvaan commented Jan 9, 2024

View reviewed changes

tinvaan changed the title ~~[WIP] Tech stack file provider~~ First draft Tech stack file provider Jan 9, 2024

tinvaan marked this pull request as ready for review January 9, 2024 12:43

tinvaan requested a review from sqs January 9, 2024 12:43

tinvaan force-pushed the providers/stackshare branch 2 times, most recently from e11983d to 4b7a1c6 Compare May 27, 2024 06:14

tinvaan added 17 commits May 27, 2024 17:44

techstack provider package.json

8fb022a

techstack file examples

8ac0464

ts and vitest configs

31e85df

First draft provider

a91c040

Fix annotation line numbers

c43d760

VScode techstacks fixtures

ba8c06e

Minor refactor

92c4b2c

node polyfills in vite config

6381039

Extend capabilities to .jsx and .tsx

e9c353e

Use fetch API instead of filesystem module

ddfda43

async annotations

b261aa5

Adapt tests to new signature

c77362f

Clear duplicate techstack yml from vscode workspace

3bf1e6d

Optimize node polyfills in vite configs

db146db

include techstack provider in web playground

f4bd238

Do not filter out blank lines

7328adf

Lines not matching import regex are needed to preserve line numbers.

Provide links to website or package homepage

69266f0

tinvaan added 12 commits May 27, 2024 17:44

Fix tests: browser and server compatible read operations

8a8242f

Check if window is undefined

894faf6

Add symlinked techstack.yml and remove node polyfills for web playground

c293299

Use absolute file uri when loading techstack settings

2353c3c

Remove node polyfills from techstack provider

22ed874

Add techstack provider to list of integrations

f9e0b49

Place techstack.yml symlink in vscode test workspace

da824c9

Minor filename pattern refactor

5cffa57

Add README

a56eb6a

Fix provider implementation

67145a3

Update README

d1df669

s/OpenCodeGraph/OpenCtx

Bundle command

85917f9

tinvaan force-pushed the providers/stackshare branch from 4b7a1c6 to 4d90f8f Compare May 27, 2024 10:14

tinvaan added 2 commits May 27, 2024 18:41

Update tests

2b128d1

Modify tsconfig settings

590a4ea

tinvaan force-pushed the providers/stackshare branch from 4d90f8f to 590a4ea Compare May 27, 2024 10:54

tinvaan added 2 commits May 27, 2024 21:45

tsc bundle force rebuild

3f4b02f

Refactor++

ecec6ab

Conversation

tinvaan commented Jan 5, 2024

Uh oh!

sqs commented Jan 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tinvaan commented Jan 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tinvaan commented Jan 7, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tinvaan commented Jan 7, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sqs left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sqs Jan 8, 2024

Choose a reason for hiding this comment

Uh oh!

sqs Jan 8, 2024

Choose a reason for hiding this comment

Uh oh!

tinvaan Jan 8, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sqs Jan 8, 2024

Choose a reason for hiding this comment

Uh oh!

tinvaan Jan 8, 2024

Choose a reason for hiding this comment

Uh oh!

tinvaan Jan 9, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tinvaan Jan 9, 2024

Choose a reason for hiding this comment

Uh oh!

sqs Jan 18, 2024

Choose a reason for hiding this comment

Uh oh!

sqs commented Jan 22, 2024

Uh oh!

tinvaan commented Jan 23, 2024

Uh oh!

sqs commented May 28, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

sqs commented Jan 6, 2024 •

edited

Loading

tinvaan commented Jan 6, 2024 •

edited

Loading

tinvaan commented Jan 7, 2024 •

edited

Loading

tinvaan commented Jan 7, 2024 •

edited

Loading

tinvaan Jan 9, 2024 •

edited

Loading