Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: proxy mode [sql changed] #571

Merged
merged 53 commits into from
Oct 13, 2024
Merged
Changes from 1 commit
Commits
Show all changes
53 commits
Select commit Hold shift + click to select a range
151369f
feat: add proxy service.
hezhengxu2018 Jun 20, 2023
8899223
feat: create task when proxy.
hezhengxu2018 Jun 23, 2023
7480e60
fix: revert npmRegistry.
hezhengxu2018 Jul 3, 2023
f3ef419
fix: npm registry adapter.
hezhengxu2018 Jul 3, 2023
03b10cc
feat: init proxy mode sql.
hezhengxu2018 Jul 9, 2023
be7c244
fix: not allow check update.
hezhengxu2018 Jul 7, 2023
c05faa7
feat: init proxy file repo.
hezhengxu2018 Jul 11, 2023
ce74646
fix: add config limit.
hezhengxu2018 Jul 12, 2023
09eedcc
refactor: rename function.
hezhengxu2018 Jul 13, 2023
c264260
feat: init schedule.
hezhengxu2018 Jul 14, 2023
1529ba9
feat: init task.
hezhengxu2018 Jul 22, 2023
d7e6c8c
feat: init task type.
hezhengxu2018 Jul 26, 2023
5fd1b14
feat: rename model.
hezhengxu2018 Jul 31, 2023
ee6beac
wip: proxy contorller.
hezhengxu2018 Aug 3, 2023
c50b0a8
wip: modify entity.
hezhengxu2018 Aug 3, 2023
186e60c
feat: realize basic func.
hezhengxu2018 Aug 4, 2023
02b30f2
fix: fix type.
hezhengxu2018 Aug 4, 2023
c3fb522
fix: fix intarval.
hezhengxu2018 Aug 7, 2023
7025d4d
test: add service test.
hezhengxu2018 Aug 12, 2023
9d2d231
test: fix test
hezhengxu2018 Aug 12, 2023
68cf8c9
test: add schedule & controller test.
hezhengxu2018 Aug 13, 2023
8fc57ec
test: fix test.
hezhengxu2018 Aug 16, 2023
aae817c
style: rename sql file.
hezhengxu2018 Oct 21, 2023
af7007d
fix: use buffer instead of TextDecoder.
hezhengxu2018 Oct 21, 2023
0776afc
feat: support clear cache.
hezhengxu2018 Dec 12, 2023
bb6159a
test: add proxy cache controller test.
hezhengxu2018 Dec 18, 2023
393d260
fix: fix update proxy cache interval
hezhengxu2018 Dec 19, 2023
fca9459
fix: unnecessary slash.
hezhengxu2018 Dec 23, 2023
ea40dba
chore: rename constant, improve type infer.
hezhengxu2018 Dec 23, 2023
dbe07d7
feat: use proxy instead of downloadToTempfile
hezhengxu2018 Dec 23, 2023
4435779
feat: add remote auth token.
hezhengxu2018 Dec 24, 2023
77285f3
feat: try remove cache dir.
hezhengxu2018 Dec 24, 2023
c5bc58d
feat: improve perf.
hezhengxu2018 Dec 29, 2023
0576532
test: improve test.
hezhengxu2018 Dec 29, 2023
e59ab0d
fix: use run in background.
hezhengxu2018 Jan 5, 2024
f2f85fe
fix: rename constant
hezhengxu2018 Mar 30, 2024
1d5a9f7
fix: rollback to temp tgz file.
hezhengxu2018 Apr 20, 2024
9526880
fix: remove unused value.
hezhengxu2018 Apr 20, 2024
42bff6a
feat: use stream instead of temp file.
hezhengxu2018 May 2, 2024
09a9304
refactor: package version controller in proxy mode.
hezhengxu2018 May 5, 2024
6e11691
refactor: package controller & package version controller.
hezhengxu2018 May 6, 2024
42c5f97
test: fix proxy controller test.
hezhengxu2018 May 31, 2024
176d336
fix: fix error proxy header.
hezhengxu2018 Jun 1, 2024
8e64b36
test: fix download package version tar controller.
hezhengxu2018 Jun 11, 2024
ca5b91c
refactor: rename function
hezhengxu2018 Jun 19, 2024
fe1c16c
fix: set version default value to null
hezhengxu2018 Jul 14, 2024
19b348d
test: improve unit test cov.
hezhengxu2018 Jul 24, 2024
f2c84ed
test: improve unit test cov.
hezhengxu2018 Jul 24, 2024
732be72
refactor: rename function
hezhengxu2018 Jul 27, 2024
63d610b
fix: convert stream
hezhengxu2018 Aug 29, 2024
ce71b9f
fix: stream type error.
hezhengxu2018 Sep 5, 2024
1ce994c
fix: enlarge retry times
hezhengxu2018 Sep 18, 2024
b07a17a
Rename 3.47.0.sql to 3.67.0.sql
fengmk2 Oct 13, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
test: fix test.
hezhengxu2018 committed Jul 14, 2024
commit 8fc57ec7ba24502af51f6882b3348c8d0238857c
9 changes: 4 additions & 5 deletions app/core/service/ProxyCacheService.ts
Original file line number Diff line number Diff line change
@@ -27,7 +27,6 @@ export function isPkgManifest(fileType: DIST_NAMES) {
}

type GetSourceManifestAndCacheReturnType<T> = {
storeKey: string,
proxyBytes: Buffer,
manifest: T extends DIST_NAMES.ABBREVIATED | DIST_NAMES.MANIFEST ? AbbreviatedPackageJSONType | PackageJSONType :
T extends DIST_NAMES.FULL_MANIFESTS | DIST_NAMES.ABBREVIATED_MANIFESTS ? AbbreviatedPackageManifestType|PackageManifestType : never;
@@ -86,7 +85,7 @@ export class ProxyCacheService extends AbstractService {
const { manifest } = await this.getSourceManifestAndCache<typeof fileType>(fullname, fileType);
const cachedFiles = ProxyCache.create({ fullname, fileType });
await this.proxyCacheRepository.saveProxyCache(cachedFiles);
hezhengxu2018 marked this conversation as resolved.
Show resolved Hide resolved
return manifest as AbbreviatedPackageManifestType|PackageManifestType;
return manifest;
}

// used by GET /:fullname/:versionOrTag
@@ -118,7 +117,7 @@ export class ProxyCacheService extends AbstractService {
const { manifest } = await this.getSourceManifestAndCache(fullname, fileType, versionOrTag);
const cachedFiles = ProxyCache.create({ fullname, fileType, version });
await this.proxyCacheRepository.saveProxyCache(cachedFiles);
return manifest as AbbreviatedPackageJSONType|PackageJSONType;
return manifest;
}

async getSourceManifestAndCache<T extends DIST_NAMES>(fullname:string, fileType: T, versionOrTag?:string): Promise<GetSourceManifestAndCacheReturnType<T>> {
@@ -166,15 +165,15 @@ export class ProxyCacheService extends AbstractService {
}
}
const proxyBytes = Buffer.from(JSON.stringify(manifest));
let storeKey;
let storeKey: string;
if (isPkgManifest(fileType)) {
storeKey = `/${PROXY_MODE_CACHED_PACKAGE_DIR_NAME}/${fullname}/${fileType}`;
} else {
const version = manifest.version;
storeKey = `/${PROXY_MODE_CACHED_PACKAGE_DIR_NAME}/${fullname}/${version}/${fileType}`;
}
await this.nfsAdapter.uploadBytes(storeKey, proxyBytes);
return { storeKey, proxyBytes, manifest };
return { proxyBytes, manifest };
}

public async createTask(targetName: string, options: UpdateProxyCacheTaskOptions): Promise<CreateUpdateProxyCacheTask> {
13 changes: 4 additions & 9 deletions test/core/service/ProxyCacheService/index.test.ts
Original file line number Diff line number Diff line change
@@ -49,12 +49,11 @@ describe('test/core/service/ProxyCacheService/index.test.ts', () => {
data,
};
});
const { manifest, storeKey } = await proxyCacheService.getSourceManifestAndCache('foobar', DIST_NAMES.FULL_MANIFESTS);
const { manifest } = await proxyCacheService.getSourceManifestAndCache('foobar', DIST_NAMES.FULL_MANIFESTS);
const versionArr = Object.values(manifest.versions);
for (const i of versionArr) {
assert(i.dist.tarball.includes('http://localhost:7001'));
}
assert.equal(storeKey, `/${PROXY_MODE_CACHED_PACKAGE_DIR_NAME}/foobar/${DIST_NAMES.FULL_MANIFESTS}`);
});

it('should get abbreviated package manifest', async () => {
@@ -65,12 +64,11 @@ describe('test/core/service/ProxyCacheService/index.test.ts', () => {
data,
};
});
const { manifest, storeKey } = await proxyCacheService.getSourceManifestAndCache('foobar', DIST_NAMES.ABBREVIATED_MANIFESTS);
const { manifest } = await proxyCacheService.getSourceManifestAndCache('foobar', DIST_NAMES.ABBREVIATED_MANIFESTS);
const versionArr = Object.values(manifest.versions);
for (const i of versionArr) {
assert(i.dist.tarball.includes('http://localhost:7001'));
}
assert.equal(storeKey, `/${PROXY_MODE_CACHED_PACKAGE_DIR_NAME}/foobar/${DIST_NAMES.ABBREVIATED_MANIFESTS}`);
});


@@ -82,10 +80,9 @@ describe('test/core/service/ProxyCacheService/index.test.ts', () => {
data,
};
});
const { manifest, storeKey } = await proxyCacheService.getSourceManifestAndCache('foobar', DIST_NAMES.MANIFEST, '1.0.0');
const { manifest } = await proxyCacheService.getSourceManifestAndCache('foobar', DIST_NAMES.MANIFEST, '1.0.0');
assert(manifest.dist);
assert(manifest.dist.tarball.includes('http://localhost:7001'));
assert.equal(storeKey, `/${PROXY_MODE_CACHED_PACKAGE_DIR_NAME}/foobar/1.0.0/${DIST_NAMES.MANIFEST}`);
});

it('should get abbreviated package version manifest', async () => {
@@ -96,18 +93,16 @@ describe('test/core/service/ProxyCacheService/index.test.ts', () => {
data,
};
});
const { manifest, storeKey } = await proxyCacheService.getSourceManifestAndCache('foobar', DIST_NAMES.ABBREVIATED, '1.0.0');
const { manifest } = await proxyCacheService.getSourceManifestAndCache('foobar', DIST_NAMES.ABBREVIATED, '1.0.0');
assert(manifest.dist);
assert(manifest.dist.tarball.includes('http://localhost:7001'));
assert.equal(storeKey, `/${PROXY_MODE_CACHED_PACKAGE_DIR_NAME}/foobar/1.0.0/${DIST_NAMES.ABBREVIATED}`);
});
});

describe('getPackageManifest()', () => {
it('should invoke getSourceManifestAndCache first.', async () => {
mock(proxyCacheService, 'getSourceManifestAndCache', async () => {
return {
storeKey: `/${PROXY_MODE_CACHED_PACKAGE_DIR_NAME}/foobar/${DIST_NAMES.FULL_MANIFESTS}`,
manifest: { name: 'mock info' },
};
});
5 changes: 2 additions & 3 deletions test/port/controller/package/ShowPackageController.test.ts
Original file line number Diff line number Diff line change
@@ -878,9 +878,8 @@ describe('test/port/controller/package/ShowPackageController.test.ts', () => {
.set('user-agent', publisher.ua + ' node/16.0.0')
.set('Accept', 'application/vnd.npm.install-v1+json');
assert(res.status === 200);
assert(res.headers.location === app.config.cnpmcore.registry);
assert(res.body.data.description === 'cnpmcore mock json');
assert(res.body.data.versions['1.0.0'].dist.tarball.includes(app.config.cnpmcore.registry));
assert(res.body.description === 'cnpmcore mock json');
assert(res.body.versions['1.0.0'].dist.tarball.includes(app.config.cnpmcore.registry));
});
});
});
Original file line number Diff line number Diff line change
@@ -383,8 +383,7 @@ describe('test/port/controller/package/ShowPackageVersionController.test.ts', ()
.set('user-agent', publisher.ua + ' node/16.0.0')
.set('Accept', 'application/vnd.npm.install-v1+json');
assert(res.status === 200);
assert(res.headers.location === app.config.cnpmcore.registry);
assert(res.body.data.dist.tarball.includes(app.config.cnpmcore.registry));
assert(res.body.dist.tarball.includes(app.config.cnpmcore.registry));
});
});
});