diff --git a/CHANGELOG.md b/CHANGELOG.md index c263b87e1..2b7681cc9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,158 +2,187 @@ ## 0.11 (Jan 19, 2024) -* `UnpackValue` is no longer implemented for `Vec`. Instead, `UnpackList`, `UnpackTuple` or `UnpackListOrTuple` can be used. -* Added `call_stack()` function. -* Removed `experimental_regex()` function. -* Switched from `anyhow::Error` and `Diagnostic` to `starlark::Error` in many APIs. -* Fix `repr()` for enums. -* Various improvements to [DAP](https://microsoft.github.io/debug-adapter-protocol/) support. -* Add a few more traits and functions to `SortedMap`. -* Disable the legacy type representation, types as strings such as `"bool"` are no longer supported. See the revised [typing documentation](https://github.com/facebookexperimental/starlark-rust/blob/main/docs/types.md). +- `UnpackValue` is no longer implemented for `Vec`. Instead, `UnpackList`, + `UnpackTuple` or `UnpackListOrTuple` can be used. +- Added `call_stack()` function. +- Removed `experimental_regex()` function. +- Switched from `anyhow::Error` and `Diagnostic` to `starlark::Error` in many + APIs. +- Fix `repr()` for enums. +- Various improvements to + [DAP](https://microsoft.github.io/debug-adapter-protocol/) support. +- Add a few more traits and functions to `SortedMap`. +- Disable the legacy type representation, types as strings such as `"bool"` are + no longer supported. See the revised + [typing documentation](https://github.com/facebookexperimental/starlark-rust/blob/main/docs/types.md). ## 0.10.0 (Oct 16, 2023) -There were over eight hundred commits since the last release, with the main areas of change including: +There were over eight hundred commits since the last release, with the main +areas of change including: -* Make it work on 32 bit architectures, including WASM. -* Introduce `starlark_syntax`, `starlark_lsp`, `starlark_bin` crates. -* Expose the AST through `starlark_syntax`. -* No longer require nightly Rust compiler. -* Change how types are represented, e.g. `{str.type, ""}` becomes `dict[str, typing.Any]`. -* Add static type checking options. -* Support f-strings extension. +- Make it work on 32 bit architectures, including WASM. +- Introduce `starlark_syntax`, `starlark_lsp`, `starlark_bin` crates. +- Expose the AST through `starlark_syntax`. +- No longer require nightly Rust compiler. +- Change how types are represented, e.g. `{str.type, ""}` becomes + `dict[str, typing.Any]`. +- Add static type checking options. +- Support f-strings extension. ## 0.9.0 (June 5, 2023) -There were over one thousand commits since the last release, with the main areas of change including: - -* Add `Module.names` to get the symbols in a module. -* Change iterators on `Value` to support reduced stack space and enable future garbage collection, removing `iterate_collect` and `with_iterator`. -* Improve DAP (debugger) support. -* Improve the performance and fix a few bugs in `%` string formatting. -* Introduce `StarlarkTypeRepr` as a trait for things that have a type representation. -* Optimise and increase standards conformance of the Starlark standard library. -* Rework how `List`/`Dict` are specified, making their internals private and introducing `ListRef`/`DictRef` for pointers to them. -* Add `AllocStruct`, `AllocList` and `AllocDict` as helpers to allocate the various types more efficiently. -* Remove the `dedupe` Starlark function. -* Integrate with the `allocative` crate to support the `Allocative` trait. -* Add more `Trace` implementations. -* Support type annotations on assignments. -* Make `Module.loads` also return the `FileSpan`. -* Add `FrozenModule.get_option`. -* Add support for integers greater than 32 bits. -* Move to a separate `starlark_map` crate for `SmallMap` and optimise it. -* Require the `starlark_type!` macro. -* Change `Module.extra_v` to be `Value<'v>`. -* Expose `FrozenHeap` from `Freezer` and `GlobalsBuilder`. -* Remove `Value::new_int`. -* In `starlark_module` functions must be more explicit - return `anyhow::Result`, have an explicit `'v` parameter, explicit `heap`/`eval` arguments. -* Add `StringValueLike` to cover string types that are values. -* Change the `json` extension method to be `json.encode` and `json.decode`. -* Support stable Rust. -* Inline all the relevant pieces of the `gazebo` library. -* Change all `starlark_module` annotations to be `#[starlark(...)]`, `#[starlark(require = named)]` for name-only parameters. -* Many performance optimisations. -* Change the API for profiling and add new profiling features. -* Make the `starlark` binary spawn the REPL by default. -* Many improvements and API changes to the documentation generation code. -* Many improvements to the LSP code. -* Add a `Dialect` option to enable types. +There were over one thousand commits since the last release, with the main areas +of change including: + +- Add `Module.names` to get the symbols in a module. +- Change iterators on `Value` to support reduced stack space and enable future + garbage collection, removing `iterate_collect` and `with_iterator`. +- Improve DAP (debugger) support. +- Improve the performance and fix a few bugs in `%` string formatting. +- Introduce `StarlarkTypeRepr` as a trait for things that have a type + representation. +- Optimise and increase standards conformance of the Starlark standard library. +- Rework how `List`/`Dict` are specified, making their internals private and + introducing `ListRef`/`DictRef` for pointers to them. +- Add `AllocStruct`, `AllocList` and `AllocDict` as helpers to allocate the + various types more efficiently. +- Remove the `dedupe` Starlark function. +- Integrate with the `allocative` crate to support the `Allocative` trait. +- Add more `Trace` implementations. +- Support type annotations on assignments. +- Make `Module.loads` also return the `FileSpan`. +- Add `FrozenModule.get_option`. +- Add support for integers greater than 32 bits. +- Move to a separate `starlark_map` crate for `SmallMap` and optimise it. +- Require the `starlark_type!` macro. +- Change `Module.extra_v` to be `Value<'v>`. +- Expose `FrozenHeap` from `Freezer` and `GlobalsBuilder`. +- Remove `Value::new_int`. +- In `starlark_module` functions must be more explicit - return + `anyhow::Result`, have an explicit `'v` parameter, explicit `heap`/`eval` + arguments. +- Add `StringValueLike` to cover string types that are values. +- Change the `json` extension method to be `json.encode` and `json.decode`. +- Support stable Rust. +- Inline all the relevant pieces of the `gazebo` library. +- Change all `starlark_module` annotations to be `#[starlark(...)]`, + `#[starlark(require = named)]` for name-only parameters. +- Many performance optimisations. +- Change the API for profiling and add new profiling features. +- Make the `starlark` binary spawn the REPL by default. +- Many improvements and API changes to the documentation generation code. +- Many improvements to the LSP code. +- Add a `Dialect` option to enable types. ## 0.8.0 (May 9, 2022) -* Redo `StringValue` and `FrozenStringValue` as aliases for `ValueTyped` and `FrozenValueTyped` respectively. -* Add more trait implementations for `ValueTyped`. -* Format call stack reports more like Python. -* Implement `FromStr` for `ProfileMode`. -* Impelement `Eq` for `BorrowHashed` . -* Minor optimisations throughout. -* Better error messages, e.g for `1 << -2`. -* Add `StarlarkValue::bit_not` for the bit-negation operator. -* In the REPL print non-`None` values. -* Add `get_index` to `SmallSet`. -* Add a `Regex` type - note this is likely to change API in the next release and is not considered stable. -* Upgrade dependencies. +- Redo `StringValue` and `FrozenStringValue` as aliases for + `ValueTyped` and `FrozenValueTyped` respectively. +- Add more trait implementations for `ValueTyped`. +- Format call stack reports more like Python. +- Implement `FromStr` for `ProfileMode`. +- Impelement `Eq` for `BorrowHashed` . +- Minor optimisations throughout. +- Better error messages, e.g for `1 << -2`. +- Add `StarlarkValue::bit_not` for the bit-negation operator. +- In the REPL print non-`None` values. +- Add `get_index` to `SmallSet`. +- Add a `Regex` type - note this is likely to change API in the next release and + is not considered stable. +- Upgrade dependencies. ## 0.7.1 (April 13, 2022) -* Add `MethodsBuilder::alloc`. +- Add `MethodsBuilder::alloc`. ## 0.7.0 (March 22, 2022) -There have been many changes since the last release, focused on performance and features. These changes caused a number of API changes and behavioural changes, the most significant of which are listed below. - -* Requires all Starlark types support `Serialize`, and provide `derive(NoSerialize)` to easily add a failing `Serialize`. -* Remove `collection_json`. -* Support the latest `gazebo` dependency. -* `SmallSet` now returns an `ExactSizeIterator`. -* Many optimisations, particularly around strings. -* Many optimisations to the bytecode compiler. -* Rename functions for working with constant strings. -* Add `StringValue` type, for `Value`s which are known to be strings. -* Remove `SimpleValue`. -* Remove the `Span` type. -* Rename `SmallHashResult` to `SmallHashValue`. -* Improve error messages on the wrong number of positional arguments. -* Remove some float-related functions from the API. -* Adjust the profiling API to make it more modular. -* Add support for validators when freezing. +There have been many changes since the last release, focused on performance and +features. These changes caused a number of API changes and behavioural changes, +the most significant of which are listed below. + +- Requires all Starlark types support `Serialize`, and provide + `derive(NoSerialize)` to easily add a failing `Serialize`. +- Remove `collection_json`. +- Support the latest `gazebo` dependency. +- `SmallSet` now returns an `ExactSizeIterator`. +- Many optimisations, particularly around strings. +- Many optimisations to the bytecode compiler. +- Rename functions for working with constant strings. +- Add `StringValue` type, for `Value`s which are known to be strings. +- Remove `SimpleValue`. +- Remove the `Span` type. +- Rename `SmallHashResult` to `SmallHashValue`. +- Improve error messages on the wrong number of positional arguments. +- Remove some float-related functions from the API. +- Adjust the profiling API to make it more modular. +- Add support for validators when freezing. ## 0.6.0 (November 22, 2021) -There have been many changes since the last release, focused on performance, documentation, type safety and profiling. These changes caused a number of API changes and behavioural changes, the most significant of which are listed below. - -* Support for newer versions of `anyhow`. -* Some error messages contain "did you mean" suggestions. -* Addition of a bytecode interpreter, with associated performance gains. -* Constant propagation and speculative execution during compilation. -* Removed mutability around the file loader and `set_loader`. -* Several new forms of profiling, making use of the new `extra_memory` function. -* Improved errors from derivations. -* Changes around function invocation, in particular `Arguments` is now opaque. -* Changes around `ConstFrozenString`, which is now `StarlarkStrN`. -* Add `OwnedFrozenValue::owner`. -* Add `derive` support for `Freeze`. -* Add more Starlark typed wrappers, such as `StringValue` and `ValueTyped`. -* Make tuples and lists opaque types, with new functions for allocating them (e.g. `alloc_tuple`). -* Make all Starlark types implement `Display` in preference to `collect_repr`. -* Support for documentation annotations on all types. +There have been many changes since the last release, focused on performance, +documentation, type safety and profiling. These changes caused a number of API +changes and behavioural changes, the most significant of which are listed below. + +- Support for newer versions of `anyhow`. +- Some error messages contain "did you mean" suggestions. +- Addition of a bytecode interpreter, with associated performance gains. +- Constant propagation and speculative execution during compilation. +- Removed mutability around the file loader and `set_loader`. +- Several new forms of profiling, making use of the new `extra_memory` function. +- Improved errors from derivations. +- Changes around function invocation, in particular `Arguments` is now opaque. +- Changes around `ConstFrozenString`, which is now `StarlarkStrN`. +- Add `OwnedFrozenValue::owner`. +- Add `derive` support for `Freeze`. +- Add more Starlark typed wrappers, such as `StringValue` and `ValueTyped`. +- Make tuples and lists opaque types, with new functions for allocating them + (e.g. `alloc_tuple`). +- Make all Starlark types implement `Display` in preference to `collect_repr`. +- Support for documentation annotations on all types. ## 0.5.0 (August 26, 2021) -There have been many changes since the last release, primarily focused on performance (up to 100x in some benchmarks). These changes caused a number of API changes, the most significant of which are listed below. - -* Rename the `starlark_module` crate to `starlark_derive`. -* Rename the `walk` methods to `trace` to align to standard GC literature. -* Add `derive` for `Trace`. -* Add `StarlarkAttrs` derivation and scheme. -* Initial start of documentation generation (still unstable). -* More complete `SmallMap` API. -* Three profiling modes, heap, flame and statement. -* Changes to `invoke` to take an `Arguments` structure. -* Changed to iteration APIs. -* Many semantic improvements to non-ASCII strings. -* Refinements to types and how they work. -* Mark a few additional APIs as `unsafe`. -* Use the `gazebo` `Coerce` trait extensively, in particular required for some of the `starlark_value` macros. -* Delete `dict.copy` and `list.copy`, since they aren't in the Starlark spec. -* `UnpackValue` no longer takes a `heap` argument. +There have been many changes since the last release, primarily focused on +performance (up to 100x in some benchmarks). These changes caused a number of +API changes, the most significant of which are listed below. + +- Rename the `starlark_module` crate to `starlark_derive`. +- Rename the `walk` methods to `trace` to align to standard GC literature. +- Add `derive` for `Trace`. +- Add `StarlarkAttrs` derivation and scheme. +- Initial start of documentation generation (still unstable). +- More complete `SmallMap` API. +- Three profiling modes, heap, flame and statement. +- Changes to `invoke` to take an `Arguments` structure. +- Changed to iteration APIs. +- Many semantic improvements to non-ASCII strings. +- Refinements to types and how they work. +- Mark a few additional APIs as `unsafe`. +- Use the `gazebo` `Coerce` trait extensively, in particular required for some + of the `starlark_value` macros. +- Delete `dict.copy` and `list.copy`, since they aren't in the Starlark spec. +- `UnpackValue` no longer takes a `heap` argument. ## 0.4.0 (April 6, 2021) -* Change maintainer to Facebook. -* Move repo to https://github.com/facebookexperimental/starlark-rust. -* Switch to a garbage collector. -* Add `#[starlark_module]` proc-macro. -* Significant rewrite of most code, changing most APIs. +- Change maintainer to Facebook. +- Move repo to https://github.com/facebookexperimental/starlark-rust. +- Switch to a garbage collector. +- Add `#[starlark_module]` proc-macro. +- Significant rewrite of most code, changing most APIs. ## 0.3.2 -* Commits and tag exist in https://github.com/indygreg/starlark-rust. -* Changed dependency versions from `X.Y.Z` to `X.Y` to allow adopting newer point releases. -* lalrpop crate upgraded from 0.16 to 0.19 and code ported to enable building on Rust 1.56+. -* Fixed compiler warnings on Rust 1.56+ related to semicolons in macro expansions. +- Commits and tag exist in https://github.com/indygreg/starlark-rust. +- Changed dependency versions from `X.Y.Z` to `X.Y` to allow adopting newer + point releases. +- lalrpop crate upgraded from 0.16 to 0.19 and code ported to enable building on + Rust 1.56+. +- Fixed compiler warnings on Rust 1.56+ related to semicolons in macro + expansions. ## 0.3.1 and before -* The code was developed at https://github.com/google/starlark-rust. +- The code was developed at https://github.com/google/starlark-rust. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index fae6547af..4946df476 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,12 +1,13 @@ # Contributing to Starlark -We want to make contributing to this project as easy and transparent as possible. +We want to make contributing to this project as easy and transparent as +possible. ## Our Development Process -Starlark is currently developed in Facebook's internal repositories and then exported -out to GitHub by a Facebook team member; however, we invite you to submit pull -requests as described below. +Starlark is currently developed in Facebook's internal repositories and then +exported out to GitHub by a Facebook team member; however, we invite you to +submit pull requests as described below. ## Pull Requests @@ -41,5 +42,5 @@ Follow the automatic `rust fmt` configuration. ## License -By contributing to Starlark, you agree that your contributions will be -licensed under the [LICENSE](LICENSE) file in the root directory of this source tree. +By contributing to Starlark, you agree that your contributions will be licensed +under the [LICENSE](LICENSE) file in the root directory of this source tree. diff --git a/README.md b/README.md index f2c6198f5..4d6a6ad36 100644 --- a/README.md +++ b/README.md @@ -6,11 +6,25 @@ [![docs.rs availability](https://img.shields.io/docsrs/starlark?label=docs.rs)](https://docs.rs/starlark/) [![Build status](https://img.shields.io/github/actions/workflow/status/facebookexperimental/starlark-rust/ci.yml?branch=main)](https://github.com/facebookexperimental/starlark-rust/actions) -There are several copies of this repo on GitHub, [facebookexperimental/starlark-rust](https://github.com/facebookexperimental/starlark-rust) is the canonical one. - -This project provides a Rust implementation of the [Starlark language](https://github.com/bazelbuild/starlark/blob/master/spec.md). Starlark (formerly codenamed Skylark) is a deterministic language inspired by Python3, used for configuration in the build systems [Bazel](https://bazel.build), [Buck](https://buck.build) and [Buck2](https://buck2.build), of which Buck2 depends on this library. This project was originally developed [in this repo](https://github.com/google/starlark-rust), which contains a more extensive history. - -There are at least three implementations of Starlark, [one in Java](https://github.com/bazelbuild/starlark), [one in Go](https://github.com/google/starlark-go), and this one in Rust. We mostly follow the Starlark standard. If you are interested in trying out Rust Starlark, you can clone this repo and run: +There are several copies of this repo on GitHub, +[facebookexperimental/starlark-rust](https://github.com/facebookexperimental/starlark-rust) +is the canonical one. + +This project provides a Rust implementation of the +[Starlark language](https://github.com/bazelbuild/starlark/blob/master/spec.md). +Starlark (formerly codenamed Skylark) is a deterministic language inspired by +Python3, used for configuration in the build systems +[Bazel](https://bazel.build), [Buck](https://buck.build) and +[Buck2](https://buck2.build), of which Buck2 depends on this library. This +project was originally developed +[in this repo](https://github.com/google/starlark-rust), which contains a more +extensive history. + +There are at least three implementations of Starlark, +[one in Java](https://github.com/bazelbuild/starlark), +[one in Go](https://github.com/google/starlark-go), and this one in Rust. We +mostly follow the Starlark standard. If you are interested in trying out Rust +Starlark, you can clone this repo and run: ```shell $ curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh @@ -19,60 +33,110 @@ $> 1+2 3 ``` -This project was started by [Damien Martin-Guillerez](https://github.com/damienmg). Version 0.4.0 of this library changed ownership [from Google](https://github.com/google/starlark-rust) to Facebook. +This project was started by +[Damien Martin-Guillerez](https://github.com/damienmg). Version 0.4.0 of this +library changed ownership [from Google](https://github.com/google/starlark-rust) +to Facebook. ## Learn More -Read [this blog post](https://developers.facebook.com/blog/post/2021/04/08/rust-starlark-library/) for an overview of the library, the reasons behind Starlark, and how it might fit in to your project. There is also a [2 minute introductory video](https://www.youtube.com/watch?v=3kHER3KIPj4). +Read +[this blog post](https://developers.facebook.com/blog/post/2021/04/08/rust-starlark-library/) +for an overview of the library, the reasons behind Starlark, and how it might +fit in to your project. There is also a +[2 minute introductory video](https://www.youtube.com/watch?v=3kHER3KIPj4). ## Features This project features: -* Easy interoperability between Rust types and Starlark. -* Rust-friendly types, so frozen values are `Send`/`Sync`, while non-frozen values aren't. -* [Garbage collected](docs/gc.md) values allocated on [a heap](docs/heaps.md). -* Optional runtime-checked [types](docs/types.md). -* A linter, to detect code issues in Starlark. -* IDE integration in the form of [LSP](https://microsoft.github.io/language-server-protocol/). -* Extensive testing, including [fuzz testing](https://github.com/google/oss-fuzz/tree/master/projects/starlark-rust). -* [DAP](https://microsoft.github.io/debug-adapter-protocol/) support. +- Easy interoperability between Rust types and Starlark. +- Rust-friendly types, so frozen values are `Send`/`Sync`, while non-frozen + values aren't. +- [Garbage collected](docs/gc.md) values allocated on [a heap](docs/heaps.md). +- Optional runtime-checked [types](docs/types.md). +- A linter, to detect code issues in Starlark. +- IDE integration in the form of + [LSP](https://microsoft.github.io/language-server-protocol/). +- Extensive testing, including + [fuzz testing](https://github.com/google/oss-fuzz/tree/master/projects/starlark-rust). +- [DAP](https://microsoft.github.io/debug-adapter-protocol/) support. This project also has three non-goals: -* We do _not_ aim for API stability between releases, preferring to iterate quickly and refine the API as much as possible. But we do [follow SemVer](https://doc.rust-lang.org/cargo/reference/semver.html). -* We do _not_ aim for minimal dependencies, preferring to keep one package with lots of power. But if some dependencies prove tricky, we might add feature flags. +- We do _not_ aim for API stability between releases, preferring to iterate + quickly and refine the API as much as possible. But we do + [follow SemVer](https://doc.rust-lang.org/cargo/reference/semver.html). +- We do _not_ aim for minimal dependencies, preferring to keep one package with + lots of power. But if some dependencies prove tricky, we might add feature + flags. ## Components There are six components: -* `starlark_derive`, a proc-macro crate that defines the necessary macros for Starlark. This library is a dependency of `starlark` the library, which reexports all the relevant pieces, and should not be used directly. -* `starlark_map`, a library with memory-efficient ordered/unordered maps/sets and various other data structures useful in Starlark. -* `starlark_syntax`, a library with the AST of Starlark and parsing functions. Only use if you want to manipulate the AST directly. -* `starlark` the main library, with evaluator, standard library, debugger support and lots of other pieces. Projects wishing to embed Starlark in their environment (with additional types, library functions and features) will make use of this library. This library reexports the relevant pieces of `starlark_derive`, `starlark_map` and most of `starlark_syntax`. -* `starlark_lsp`, a library providing an [LSP](https://microsoft.github.io/language-server-protocol/). -* `starlark_bin` the binary, which provides interactive evaluation, IDE features and linter, exposed through a command line. Useful if you want to use vanilla Starlark (but if you do, consider Python3 instead) or as a test-bed for experimenting. Most projects will end up implementing some of this functionality themselves over the `starlark` and `starlark_lsp` libraries, incorporating their specific extra types etc. - -In particular the `starlark_bin` binary _can_ be effectively used as a linter. But for the REPL, evaluator and IDE features the `starlark_bin` binary is only aware of standard Starlark. Most Starlark embeddings supply extra functions and data types to work with domain-specific concerns, and the lack of these bindings will cause the REPL/evaluator to fail if they are used, and will give a subpar IDE experience. In most cases you should write your own binary depending on the `starlark` library, integrating your domain-specific pieces, and then using the bundled LSP functions to produce your own IDE/REPL/evaluator on top of those. You should still be able to use the [VS Code extension](vscode/README.md). +- `starlark_derive`, a proc-macro crate that defines the necessary macros for + Starlark. This library is a dependency of `starlark` the library, which + reexports all the relevant pieces, and should not be used directly. +- `starlark_map`, a library with memory-efficient ordered/unordered maps/sets + and various other data structures useful in Starlark. +- `starlark_syntax`, a library with the AST of Starlark and parsing functions. + Only use if you want to manipulate the AST directly. +- `starlark` the main library, with evaluator, standard library, debugger + support and lots of other pieces. Projects wishing to embed Starlark in their + environment (with additional types, library functions and features) will make + use of this library. This library reexports the relevant pieces of + `starlark_derive`, `starlark_map` and most of `starlark_syntax`. +- `starlark_lsp`, a library providing an + [LSP](https://microsoft.github.io/language-server-protocol/). +- `starlark_bin` the binary, which provides interactive evaluation, IDE features + and linter, exposed through a command line. Useful if you want to use vanilla + Starlark (but if you do, consider Python3 instead) or as a test-bed for + experimenting. Most projects will end up implementing some of this + functionality themselves over the `starlark` and `starlark_lsp` libraries, + incorporating their specific extra types etc. + +In particular the `starlark_bin` binary _can_ be effectively used as a linter. +But for the REPL, evaluator and IDE features the `starlark_bin` binary is only +aware of standard Starlark. Most Starlark embeddings supply extra functions and +data types to work with domain-specific concerns, and the lack of these bindings +will cause the REPL/evaluator to fail if they are used, and will give a subpar +IDE experience. In most cases you should write your own binary depending on the +`starlark` library, integrating your domain-specific pieces, and then using the +bundled LSP functions to produce your own IDE/REPL/evaluator on top of those. +You should still be able to use the [VS Code extension](vscode/README.md). ## Compatibility -In this section we outline where we don't comply with the [Starlark spec](https://github.com/bazelbuild/starlark/blob/master/spec.md). +In this section we outline where we don't comply with the +[Starlark spec](https://github.com/bazelbuild/starlark/blob/master/spec.md). -* We have plenty of extensions, e.g. type annotations, recursion, top-level `for`. -* We don't yet support later additions to Starlark, such as [bytes](https://github.com/facebookexperimental/starlark-rust/issues/4). -* In some cases creating circular data structures may lead to stack overflows. +- We have plenty of extensions, e.g. type annotations, recursion, top-level + `for`. +- We don't yet support later additions to Starlark, such as + [bytes](https://github.com/facebookexperimental/starlark-rust/issues/4). +- In some cases creating circular data structures may lead to stack overflows. ## Making a release -1. Check the [GitHub Actions](https://github.com/facebookexperimental/starlark-rust/actions) are green. -2. Update `CHANGELOG.md` with the changes since the last release. [This link](https://github.com/facebookexperimental/starlark-rust/compare/v0.4.0...main) can help (update to compare against the last release). -3. Update the version numbers of the two `Cargo.toml` files. Bump them by 0.0.1 if there are no incompatible changes, or 0.1.0 if there are. Bump the dependency in `starlark` to point at the latest `starlark_derive` version. -4. Copy the files `CHANGELOG.md`, `LICENSE` and `README.md` into each subdirectory. -5. Run `cargo publish --allow-dirty --dry-run`, then without the `--dry-run`, in each of the component directories in the [order above](#components). -6. Create a [GitHub release](https://github.com/facebookexperimental/starlark-rust/releases/new) with `v0.X.Y`, using the `starlark` version as the name. +1. Check the + [GitHub Actions](https://github.com/facebookexperimental/starlark-rust/actions) + are green. +2. Update `CHANGELOG.md` with the changes since the last release. + [This link](https://github.com/facebookexperimental/starlark-rust/compare/v0.4.0...main) + can help (update to compare against the last release). +3. Update the version numbers of the two `Cargo.toml` files. Bump them by 0.0.1 + if there are no incompatible changes, or 0.1.0 if there are. Bump the + dependency in `starlark` to point at the latest `starlark_derive` version. +4. Copy the files `CHANGELOG.md`, `LICENSE` and `README.md` into each + subdirectory. +5. Run `cargo publish --allow-dirty --dry-run`, then without the `--dry-run`, in + each of the component directories in the [order above](#components). +6. Create a + [GitHub release](https://github.com/facebookexperimental/starlark-rust/releases/new) + with `v0.X.Y`, using the `starlark` version as the name. ## License -Starlark Rust is Apache License, Version 2.0 licensed, as found in the [LICENSE](LICENSE) file. +Starlark Rust is Apache License, Version 2.0 licensed, as found in the +[LICENSE](LICENSE) file. diff --git a/allocative/README.md b/allocative/README.md index 56fe805b1..2c13bd7b2 100644 --- a/allocative/README.md +++ b/allocative/README.md @@ -1,41 +1,42 @@ # Allocative: memory profiler for Rust -This crate implements a lightweight memory profiler which allows -object traversal and memory size introspection. +This crate implements a lightweight memory profiler which allows object +traversal and memory size introspection. ## Usage `Allocative` trait (typically implemented with proc-macro) is introspectable: -`Allocative` values can be traversed and their size and sizes of referenced objects -can be collected. +`Allocative` values can be traversed and their size and sizes of referenced +objects can be collected. -This crate provides a few utilities to work with such objects, -the main of such utilities is flame graph builder which produces flame graph -(see the crate documentation) like this: +This crate provides a few utilities to work with such objects, the main of such +utilities is flame graph builder which produces flame graph (see the crate +documentation) like this: ![sample-flamegraph.png](sample-flamegraph.png) ## How it is different from other call-stack malloc profilers like jemalloc heap profiler -Allocative is not a substitute for call stack malloc profiler, -it provides a different view of memory usage. +Allocative is not a substitute for call stack malloc profiler, it provides a +different view of memory usage. Here are some differences between allocative and call-stack malloc profiler: -* Allocative requires implementation of `Allocative` trait for each type - which needs to be measured, and some setup in the program to enable it is needed -* Allocative flamegraph shows object by object tree, not by call stack -* Allocative shows gaps in allocated memory, - e.g. spare capacity of collections or too large padding in structs or enums -* Allocative allows profiling of non-malloc allocations - (for example, allocations within [bumpalo](https://github.com/fitzgen/bumpalo) bumps) -* Allocative allows profiling of memory for subset of the process data - (for example, measure the size of RPC response before serialization) +- Allocative requires implementation of `Allocative` trait for each type which + needs to be measured, and some setup in the program to enable it is needed +- Allocative flamegraph shows object by object tree, not by call stack +- Allocative shows gaps in allocated memory, e.g. spare capacity of collections + or too large padding in structs or enums +- Allocative allows profiling of non-malloc allocations (for example, + allocations within [bumpalo](https://github.com/fitzgen/bumpalo) bumps) +- Allocative allows profiling of memory for subset of the process data (for + example, measure the size of RPC response before serialization) ## Runtime overhead -When allocative is used, binary size is slightly increased due to implementations -of [`Allocative`] trait, but it has no runtime/memory overhead when it is enabled but not used. +When allocative is used, binary size is slightly increased due to +implementations of [`Allocative`] trait, but it has no runtime/memory overhead +when it is enabled but not used. ## Source code @@ -45,5 +46,5 @@ is synchronized to GitHub. The main copy is ## License -Allocative is both MIT and Apache License, Version 2.0 licensed, -as found in the [LICENSE-MIT](LICENSE-MIT) and [LICENSE-APACHE](LICENSE-APACHE) files. +Allocative is both MIT and Apache License, Version 2.0 licensed, as found in the +[LICENSE-MIT](LICENSE-MIT) and [LICENSE-APACHE](LICENSE-APACHE) files. diff --git a/docs/environment.md b/docs/environment.md index 9bd80b5d9..d527d482e 100644 --- a/docs/environment.md +++ b/docs/environment.md @@ -1,10 +1,11 @@ # Environments -:::warning -Some of the information within this page is outdated. However, the explanation of the problem, and thought process behind it, remains useful. The storage of values is similar but implemented using different types. -::: +:::warning Some of the information within this page is outdated. However, the +explanation of the problem, and thought process behind it, remains useful. The +storage of values is similar but implemented using different types. ::: -Starlark (with a nested `def`) has a series of environments that may be active during an evaluation, as illustrated in the following example: +Starlark (with a nested `def`) has a series of environments that may be active +during an evaluation, as illustrated in the following example: ```python x = [] @@ -17,21 +18,30 @@ def foo(): The above example features the following environments: -* Global environment - defining things like `list.append` -* Module environment - defining `x` -* Environment of `foo` - defining `y` -* Environment of `bar` - defining `z` +- Global environment - defining things like `list.append` +- Module environment - defining `x` +- Environment of `foo` - defining `y` +- Environment of `bar` - defining `z` -A scope can *access* variables defined above it, and often *mutate* them, but not *assign* them. +A scope can _access_ variables defined above it, and often _mutate_ them, but +not _assign_ them. To unpack that: -* From the statements inside `bar`, you can access `list.append`, `x`, `y`, and `z`. -* From inside `bar`, you can mutate the variables to be accessed with statements like `list.append(x, 1)` (which may also be termed `x.append(1)`). - * However, before this module is imported by another module, all of its exports become *frozen*, which means it isn't possible to mutate a global list, and if `foo` is called from a different module, then `x` can't be modified. -* If `bar` does `x = 1` that defines a local variable `x` in the function `bar`, shadowing the global `x`. As a consequence, you cannot assign to variables defined in an outer scope. - -Note that assignment *after*, or even *in* non-executed conditional branches, introduces a local variable. +- From the statements inside `bar`, you can access `list.append`, `x`, `y`, and + `z`. +- From inside `bar`, you can mutate the variables to be accessed with statements + like `list.append(x, 1)` (which may also be termed `x.append(1)`). + - However, before this module is imported by another module, all of its + exports become _frozen_, which means it isn't possible to mutate a global + list, and if `foo` is called from a different module, then `x` can't be + modified. +- If `bar` does `x = 1` that defines a local variable `x` in the function `bar`, + shadowing the global `x`. As a consequence, you cannot assign to variables + defined in an outer scope. + +Note that assignment _after_, or even _in_ non-executed conditional branches, +introduces a local variable. For example: @@ -43,21 +53,28 @@ def f(): x = 2 ``` -In the above code, on executing `f()`, it would complain that `x` is referenced before assignment, as the assignment `x = 2` makes `x` a local variable. +In the above code, on executing `f()`, it would complain that `x` is referenced +before assignment, as the assignment `x = 2` makes `x` a local variable. -The rest of this document outlines the various types of environments, how they are accessed, and how they are updated. +The rest of this document outlines the various types of environments, how they +are accessed, and how they are updated. ## Global Environment -The global environment is always frozen and consists of *functions* and *type-values*. All things in the global environment are accessed by name. +The global environment is always frozen and consists of _functions_ and +_type-values_. All things in the global environment are accessed by name. -Type-values are things like `list.append`, which is used when you do either `list.append(xs, 1)` or `xs.append(1)`, assuming `xs` is of type `list`. The available methods for a type can be queried (for example, `dir(list)`). +Type-values are things like `list.append`, which is used when you do either +`list.append(xs, 1)` or `xs.append(1)`, assuming `xs` is of type `list`. The +available methods for a type can be queried (for example, `dir(list)`). There are also global functions, such as `len`, `range`, and `str`. ## Slots -To optimise evaluation, all variables are accessed by integers, which are known as 'slots'. Many variables can be converted to slots statically during compilation, and those which can't have their slot looked up by name at runtime. +To optimise evaluation, all variables are accessed by integers, which are known +as 'slots'. Many variables can be converted to slots statically during +compilation, and those which can't have their slot looked up by name at runtime. The `Slots` data type is defined as: @@ -72,25 +89,39 @@ struct FrozenSlots(Arc>>); As featured in the above code: -* A set of slots are either `Frozen`, which came from another module behind `Arc` or just normal `Slots`, which can be manipulated by the current scope (behind a `Rc`/`RefCell` for single-threaded use and mutation). -* `Vec` is accessed by the slot index. -* `Option` refers to whether the slot has been assigned yet (to detect variables referenced before assignment). +- A set of slots are either `Frozen`, which came from another module behind + `Arc` or just normal `Slots`, which can be manipulated by the current scope + (behind a `Rc`/`RefCell` for single-threaded use and mutation). +- `Vec` is accessed by the slot index. +- `Option` refers to whether the slot has been assigned yet (to detect variables + referenced before assignment). ## Module Environment -The module environment is where the module executes, namely where `x` is defined above. The module environment can have values added in the following standards-conforming ways: +The module environment is where the module executes, namely where `x` is defined +above. The module environment can have values added in the following +standards-conforming ways: -* Assignment statements (such as `x = 1` or `x += 1`). -* `For` loops (such as the `x` in `for x in []:`). -* Via the `load("a.bzl", "foo")`, which imports `foo` frozen. -* Via `def foo():`, which defines `foo` in the module environment. Whether a `def` is frozen or not, when it's executed, its local variables are not frozen. +- Assignment statements (such as `x = 1` or `x += 1`). +- `For` loops (such as the `x` in `for x in []:`). +- Via the `load("a.bzl", "foo")`, which imports `foo` frozen. +- Via `def foo():`, which defines `foo` in the module environment. Whether a + `def` is frozen or not, when it's executed, its local variables are not + frozen. -In addition, two non-standards-conforming ways of defining variables are supported: +In addition, two non-standards-conforming ways of defining variables are +supported: -* Some modules can be injected as bindings in advance. Given a module `foo` that is injected, all the bindings of `foo` will be inserted in this module as frozen. -* The function `load_symbols` injects a dictionary of bindings into the module environment. +- Some modules can be injected as bindings in advance. Given a module `foo` that + is injected, all the bindings of `foo` will be inserted in this module as + frozen. +- The function `load_symbols` injects a dictionary of bindings into the module + environment. -Note that a module has a fixed set of variables (from the standards-conforming ways), a pre-execution set (from the injections) and yet more variables at runtime (via `load_symbols`). To support that structure, the mapping from name to slot index is tracked in a struct: +Note that a module has a fixed set of variables (from the standards-conforming +ways), a pre-execution set (from the injections) and yet more variables at +runtime (via `load_symbols`). To support that structure, the mapping from name +to slot index is tracked in a struct: ```rust enum Names { @@ -100,15 +131,23 @@ enum Names { struct FrozenNames(Arc>); ``` -Each name is given an entry in the map with an increasing slot index. A name will only be assigned a slot once, reusing it thereafter. A corresponding `Slots` data type provides the values associated with those names. +Each name is given an entry in the map with an increasing slot index. A name +will only be assigned a slot once, reusing it thereafter. A corresponding +`Slots` data type provides the values associated with those names. -Importantly, the `Slots` can be extended at runtime by the `load_symbols` function. As with `Slots`, you can either share things behind an `Arc` or mutate them behind an `Rc`/`RefCell`. +Importantly, the `Slots` can be extended at runtime by the `load_symbols` +function. As with `Slots`, you can either share things behind an `Arc` or mutate +them behind an `Rc`/`RefCell`. ## Function Environment -A function can have variables introduced via assignments, `for` loops, and parameters. No additional variables can be discovered at runtime, so all names can be erased at compile time. +A function can have variables introduced via assignments, `for` loops, and +parameters. No additional variables can be discovered at runtime, so all names +can be erased at compile time. -A function can also access variables from the functions it is statically nested within, and from the variables at the root of the module. To support this structure, at runtime we pass around the context, defined as: +A function can also access variables from the functions it is statically nested +within, and from the variables at the root of the module. To support this +structure, at runtime we pass around the context, defined as: ```rust struct Context { @@ -117,13 +156,15 @@ struct Context { } ``` -The above code contains the mapping of names for the module and the slots for the module and each function. +The above code contains the mapping of names for the module and the slots for +the module and each function. -When executed, the inner-most `Slots` (at the end of `slots:`) will never be frozen, as that represents the local variables: but any other may be. +When executed, the inner-most `Slots` (at the end of `slots:`) will never be +frozen, as that represents the local variables: but any other may be. When a function value is captured in a frozen module, use `FrozenContext`: -```rust +````rust struct FrozenContext { names: FrozenNames, slots: Vec, @@ -135,21 +176,34 @@ A list comprehension can be defined as: ```python [x for x in [1,2,3]] -``` +```` In the above code: -* The statement defines a variable `x` that is immediately initialised and shadows any other variables `x` in scope. -* The variable `x` cannot be assigned to, other than in the list comprehension, as it only lives inside the comprehension and the comprehension does not permit assignment statements (only expressions). Such names are not available at the top-level, even when defined in the root of a module. +- The statement defines a variable `x` that is immediately initialised and + shadows any other variables `x` in scope. +- The variable `x` cannot be assigned to, other than in the list comprehension, + as it only lives inside the comprehension and the comprehension does not + permit assignment statements (only expressions). Such names are not available + at the top-level, even when defined in the root of a module. -List comprehensions are implemented by adding additional entries into the `Slots` data type. Even when added at the root of a module, such names are not added to `Names`. +List comprehensions are implemented by adding additional entries into the +`Slots` data type. Even when added at the root of a module, such names are not +added to `Names`. ## Optimisations There are a number of optimisations made to the scheme: -* When freezing a `Names` or `Slots` structure, it's important to only freeze a particular mutable variant once, or you duplicate memory unnecessarily. Therefore, the `Slots` to be `Rc)>>` are augmented, and, similarly, the `Names`. - * When `freeze` is called, the original value is consumed, and the `Some` variant is added. - * **Note**: it is unsafe to ever access the slots after the `freeze`. -* Programs can only assign to the inner-most `Slots`, and that slots must always be mutable. Therefore, define a local `Slots` that is always mutable, and a separate AST node for referring to it. - * For modules, it is important that this mutable local `Slots` is *also* in scope since the scope is used to retrieve unknown variables. +- When freezing a `Names` or `Slots` structure, it's important to only freeze a + particular mutable variant once, or you duplicate memory unnecessarily. + Therefore, the `Slots` to be `Rc)>>` are + augmented, and, similarly, the `Names`. + - When `freeze` is called, the original value is consumed, and the `Some` + variant is added. + - **Note**: it is unsafe to ever access the slots after the `freeze`. +- Programs can only assign to the inner-most `Slots`, and that slots must always + be mutable. Therefore, define a local `Slots` that is always mutable, and a + separate AST node for referring to it. + - For modules, it is important that this mutable local `Slots` is _also_ in + scope since the scope is used to retrieve unknown variables. diff --git a/docs/gc.md b/docs/gc.md index 324a3a5a8..847d48354 100644 --- a/docs/gc.md +++ b/docs/gc.md @@ -2,7 +2,14 @@ This page describes a two-space garbage collector that can deal with cycles. -In Starlark, this pattern is used both when doing a real garbage collection, and when freezing. For both cases, it starts out with a memory block, which has pointers referring to things inside it, and ends up with a new memory block with equivalent pointers inside it. However, only pointers reachable from outside the original memory block are available in the new memory block. The garbage collector can deal with cyclic data structures and the time spent is proportional to the amount of live data in the heap (memory that is dropped is not even visited). +In Starlark, this pattern is used both when doing a real garbage collection, and +when freezing. For both cases, it starts out with a memory block, which has +pointers referring to things inside it, and ends up with a new memory block with +equivalent pointers inside it. However, only pointers reachable from outside the +original memory block are available in the new memory block. The garbage +collector can deal with cyclic data structures and the time spent is +proportional to the amount of live data in the heap (memory that is dropped is +not even visited). ## A worked example @@ -14,53 +21,67 @@ Y := Data("hello", X, Y) Z := Data("universe") ``` -All of `X`, `Y` and `Z` are memory locations. The `Y` memory location has both some data of its own (`"hello"`) and two pointers (`X` and `Y` itself). +All of `X`, `Y` and `Z` are memory locations. The `Y` memory location has both +some data of its own (`"hello"`) and two pointers (`X` and `Y` itself). -The pointers from outside the heap into the heap are known as *roots*. +The pointers from outside the heap into the heap are known as _roots_. -Assuming, in the above example, that `Y` is the only root, then, since `Y` is used from outside, `Y` must be moved to the new memory block. Consequently, the data `X` needs to be copied, but `Z` can be dropped. +Assuming, in the above example, that `Y` is the only root, then, since `Y` is +used from outside, `Y` must be moved to the new memory block. Consequently, the +data `X` needs to be copied, but `Z` can be dropped. Following are the required steps for using a garbage collector: -1. To copy `Y`, allocate a value in the new heap `A` with a sentinel value in it (that that sentinel is called a `Blackhole`). Then, turn `Y` into a `Forward(A)` pointer, so that if anyone else in this cycle tries to collect `Y` they immediately "forward" to the new value and the data from `Y` is grabbed so its pointers can be traversed. That results in the following: +1. To copy `Y`, allocate a value in the new heap `A` with a sentinel value in it + (that that sentinel is called a `Blackhole`). Then, turn `Y` into a + `Forward(A)` pointer, so that if anyone else in this cycle tries to collect + `Y` they immediately "forward" to the new value and the data from `Y` is + grabbed so its pointers can be traversed. That results in the following: - ```bash - X := Data("world") - Y := Forward(A) - Z := Data("universe") + ```bash + X := Data("world") + Y := Forward(A) + Z := Data("universe") - A := Blackhole - ``` + A := Blackhole + ``` - With `Data("hello", X, Y)` as the current item being processed. + With `Data("hello", X, Y)` as the current item being processed. -2. Walk the pointers of the current value, performing a garbage collection on each of them. To copy `Y`, it can be seen that `Y` points at a `Forward(A)` node, so there's no need to do anything. To copy `X`, follow the process starting at step 1, but for `X` (which ends up at `B`). Performing that move leads to the following: +2. Walk the pointers of the current value, performing a garbage collection on + each of them. To copy `Y`, it can be seen that `Y` points at a `Forward(A)` + node, so there's no need to do anything. To copy `X`, follow the process + starting at step 1, but for `X` (which ends up at `B`). Performing that move + leads to the following: - ```bash - X := Forward(B) - Y := Forward(A) - Z := Data("universe") + ```bash + X := Forward(B) + Y := Forward(A) + Z := Data("universe") - A := Blackhole - B := Data("world") - ``` + A := Blackhole + B := Data("world") + ``` -3. Replace all the pointers with the forwarded value, and write it back over the `Blackhole` in `A`. This gives the following: +3. Replace all the pointers with the forwarded value, and write it back over the + `Blackhole` in `A`. This gives the following: - ```bash - X := Forward(B) - Y := Forward(A) - Z := Data("universe") + ```bash + X := Forward(B) + Y := Forward(A) + Z := Data("universe") - A := Data("hello", B, A) - B := Data("world") - ``` + A := Data("hello", B, A) + B := Data("world") + ``` -4. Adjust any roots pointing at `Y` to point at `A` and throw away the original heap, which produces the following: +4. Adjust any roots pointing at `Y` to point at `A` and throw away the original + heap, which produces the following: - ```bash - A := Data("hello", B, A) - B := Data("world") - ``` + ```bash + A := Data("hello", B, A) + B := Data("world") + ``` -These above four steps successfully garbage collects a cyclic data structure, while preserving the cycles and getting rid of the unused data. +These above four steps successfully garbage collects a cyclic data structure, +while preserving the cycles and getting rid of the unused data. diff --git a/docs/heaps.md b/docs/heaps.md index 22cf8497b..0af4f4ebc 100644 --- a/docs/heaps.md +++ b/docs/heaps.md @@ -4,24 +4,36 @@ In Starlark, there are three interesting heap-related points of interest: -* A `Heap` has `Value`'s allocated on it and cannot be cloned or shared. -* A `FrozenHeap` has `FrozenValue`'s allocated on it and cannot be cloned or shared. -* A `FrozenHeapRef` is a `FrozenHeap` that is now read-only and can now be cloned and shared. +- A `Heap` has `Value`'s allocated on it and cannot be cloned or shared. +- A `FrozenHeap` has `FrozenValue`'s allocated on it and cannot be cloned or + shared. +- A `FrozenHeapRef` is a `FrozenHeap` that is now read-only and can now be + cloned and shared. -A `FrozenHeapRef` keeps a heap alive. While you have a `FrozenValue`, it is important that you have either the `FrozenHeap` itself, or more usually, a `FrozenHeapRef` to it. A `FrozenHeap` may contains a set of `FrozenHeapRef`'s to keep the `FrozenHeap`s it references alive. +A `FrozenHeapRef` keeps a heap alive. While you have a `FrozenValue`, it is +important that you have either the `FrozenHeap` itself, or more usually, a +`FrozenHeapRef` to it. A `FrozenHeap` may contains a set of `FrozenHeapRef`'s to +keep the `FrozenHeap`s it references alive. ## Heap Containers Heaps are included in other data types: -* A `Module` contains a `Heap` (where normal values are allocated) and a `FrozenHeap` (stores references to other frozen heaps and has compilation constants allocated on it). The `Heap` portion is garbage collected. At the end, when you call `freeze`, `Value`'s referenced by name in the `Module` are moved to the `FrozenHeap` and then then `FrozenHeap` is sealed to produce a `FrozenHeapRef`. -* A `FrozenModule` contains a `FrozenHeapRef`. -* A `GlobalsBuilder` contains a `FrozenHeap` onto which values are allocated. -* A `Globals` contains a `FrozenHeapRef`. +- A `Module` contains a `Heap` (where normal values are allocated) and a + `FrozenHeap` (stores references to other frozen heaps and has compilation + constants allocated on it). The `Heap` portion is garbage collected. At the + end, when you call `freeze`, `Value`'s referenced by name in the `Module` are + moved to the `FrozenHeap` and then then `FrozenHeap` is sealed to produce a + `FrozenHeapRef`. +- A `FrozenModule` contains a `FrozenHeapRef`. +- A `GlobalsBuilder` contains a `FrozenHeap` onto which values are allocated. +- A `Globals` contains a `FrozenHeapRef`. ## Heap References -It is important that when a `FrozenValue` X is referenced by a `Value` or `FrozenValue` (for example, included in a list), the heap where X originates is added as a reference to the heap where the new value is being created. +It is important that when a `FrozenValue` X is referenced by a `Value` or +`FrozenValue` (for example, included in a list), the heap where X originates is +added as a reference to the heap where the new value is being created. As a concrete example in pseudo-code: @@ -40,19 +52,29 @@ In the above code, the following steps are taken: 1. Create a `FrozenHeap` then allocate something in it. 1. Turn the heap into a reference. 1. Use the allocated value `s` from `h1` when constructing a value in `h2`. -1. For that to be legal, and for the heap `h1` to not disappear while it is being allocated, it is important to call `add_reference`. +1. For that to be legal, and for the heap `h1` to not disappear while it is + being allocated, it is important to call `add_reference`. -Note that this API can only point at a `FrozenValue` from another heap, and only after that heap has been turned into a reference, so it will not be allocated in anymore. These restrictions are deliberate and mean that most programs only have one 'active heap' at a time. +Note that this API can only point at a `FrozenValue` from another heap, and only +after that heap has been turned into a reference, so it will not be allocated in +anymore. These restrictions are deliberate and mean that most programs only have +one 'active heap' at a time. Following are some places where heap references are added by Starlark: -* Before evaluation is started, a reference is added to the `Globals` from the `Module`, so it can access the global functions. -* When evaluating a `load` statement, a reference is added to the `FrozenModule` that is being loaded. -* When freezing a module, the `FrozenHeap`, in the `Module`, is moved to the `FrozenModule`, preserving the references that were added. +- Before evaluation is started, a reference is added to the `Globals` from the + `Module`, so it can access the global functions. +- When evaluating a `load` statement, a reference is added to the `FrozenModule` + that is being loaded. +- When freezing a module, the `FrozenHeap`, in the `Module`, is moved to the + `FrozenModule`, preserving the references that were added. ## `OwnedFrozenValue` -When you get a value from a `FrozenModule`, it will be a `OwnedFrozenValue`. This structure is a pair of a `FrozenHeapRef` and a `FrozenValue`, where the ref keeps the value alive. You can move that `OwnedFrozenValue` into the value of a module with code such as: +When you get a value from a `FrozenModule`, it will be a `OwnedFrozenValue`. +This structure is a pair of a `FrozenHeapRef` and a `FrozenValue`, where the ref +keeps the value alive. You can move that `OwnedFrozenValue` into the value of a +module with code such as: ```rust fn move<'v>(from: &FrozenModule, to: &'v Module) { @@ -64,9 +86,16 @@ fn move<'v>(from: &FrozenModule, to: &'v Module) { In general, you can use the `OwnedFrozenValue` in one of three ways: -* **Operate on it directly** - with methods like `unpack_i32` or `to_str`. -* **Extract it safely** - using methods like `owned_frozen_value`, which takes a `FrozenHeap` to which the heap reference is added and returns a naked `FrozenValue`. After that, it is then safe for the `FrozenHeap` you passed in to use the `FrozenValue`. - * With `owned_value`, there is lifetime checking that the right heap is passed, but with `FrozenValue`, there isn't. - * Be careful to pass the right heap, although given most programs only have one active heap at a time, it should mostly work out. -* **Extract it unsafely** - using methods `unchecked_frozen_value`, which gives you the underlying `FrozenValue` without adding any references. - * Be careful to make sure there is a good reason the `FrozenValue` remains valid. +- **Operate on it directly** - with methods like `unpack_i32` or `to_str`. +- **Extract it safely** - using methods like `owned_frozen_value`, which takes a + `FrozenHeap` to which the heap reference is added and returns a naked + `FrozenValue`. After that, it is then safe for the `FrozenHeap` you passed in + to use the `FrozenValue`. + - With `owned_value`, there is lifetime checking that the right heap is + passed, but with `FrozenValue`, there isn't. + - Be careful to pass the right heap, although given most programs only have + one active heap at a time, it should mostly work out. +- **Extract it unsafely** - using methods `unchecked_frozen_value`, which gives + you the underlying `FrozenValue` without adding any references. + - Be careful to make sure there is a good reason the `FrozenValue` remains + valid. diff --git a/docs/types.md b/docs/types.md index 009c86285..c50b20efa 100644 --- a/docs/types.md +++ b/docs/types.md @@ -1,6 +1,7 @@ # Starlark Types -The Starlark 'types' extension is highly experimental and likely to be modified in the future. +The Starlark 'types' extension is highly experimental and likely to be modified +in the future. Types can be added to function arguments, or function return types. @@ -13,46 +14,68 @@ def fib(i: int) -> int: There are moments where types can be checked: -1. At runtime, as a function is executed, when a value of the appropriate type is available. +1. At runtime, as a function is executed, when a value of the appropriate type + is available. 2. Statically, without executing anything. -3. At compile time, when the definitions of all symbols imported using `load` are available. +3. At compile time, when the definitions of all symbols imported using `load` + are available. -Currently runtime is the normal way of checking, but other systems built on Starlark (e.g. Buck2) may also perform additional types of checking. In all cases the meaning of the types is the same. +Currently runtime is the normal way of checking, but other systems built on +Starlark (e.g. Buck2) may also perform additional types of checking. In all +cases the meaning of the types is the same. -The rest of this document lays out what types mean and what type-supporting values are available (records and enums). +The rest of this document lays out what types mean and what type-supporting +values are available (records and enums). ## What does a type mean? A type is a Starlark expression that has a meaning as a type: -* When `fib(3)` is called, the *value* `3` is passed to `fib` as parameter `i`. -* When the execution of `fib` is started, the *expression* `int` is evaluated to the value of the `int` function. -* A check is then made that the value `3` matches the type represented by `int`. +- When `fib(3)` is called, the _value_ `3` is passed to `fib` as parameter `i`. +- When the execution of `fib` is started, the _expression_ `int` is evaluated to + the value of the `int` function. +- A check is then made that the value `3` matches the type represented by `int`. -If the value doesn't match, it is a runtime error. Similarly, on `return` statements, or the end of the function, a check is made that result type matches `int`. +If the value doesn't match, it is a runtime error. Similarly, on `return` +statements, or the end of the function, a check is made that result type matches +`int`. As some examples of types: -* The type `typing.Any` matches any value, with no restrictions. -* The types `int`, `bool`, `str` all represent the values produced by the respective functions. -* The type `None` represents the value `None`. -* The type `list[int]` represents a list of `int` types, e.g. `list[typing.Any]` represents a list containing any types. -* The type `dict[int, bool]` represents a dictionary with `int` keys and `bool` values. -* The type `tuple[int, bool, str]` represents a tuple of arity 3 with components being `int`, `bool` and `str`. -* The type `tuple[int, ...]` represents a tuple of unknown arity where all the components are of type `int`. -* The type `int | bool` represents a value that is either an `int` or a `bool`. -* The type `typing.Callable` represents something that can be called as a function. -* The type `typing.Iterable` represents something that can be iterated on. -* The type `typing.Never` represents a type with no valid values - e.g. the result of `fail` is `typing.Never` as the return value of `fail` can never be observed, given the program terminates. +- The type `typing.Any` matches any value, with no restrictions. +- The types `int`, `bool`, `str` all represent the values produced by the + respective functions. +- The type `None` represents the value `None`. +- The type `list[int]` represents a list of `int` types, e.g. `list[typing.Any]` + represents a list containing any types. +- The type `dict[int, bool]` represents a dictionary with `int` keys and `bool` + values. +- The type `tuple[int, bool, str]` represents a tuple of arity 3 with components + being `int`, `bool` and `str`. +- The type `tuple[int, ...]` represents a tuple of unknown arity where all the + components are of type `int`. +- The type `int | bool` represents a value that is either an `int` or a `bool`. +- The type `typing.Callable` represents something that can be called as a + function. +- The type `typing.Iterable` represents something that can be iterated on. +- The type `typing.Never` represents a type with no valid values - e.g. the + result of `fail` is `typing.Never` as the return value of `fail` can never be + observed, given the program terminates. The goals of this type system are: -* Reuse the existing machinery of Starlark as much as possible, avoiding inventing a special class of type values. As a consequence, any optimisations for values like string/list are reused. -* Provide a pleasing syntax. -* Some degree of compatibility with Python, which allows types as expressions in the same places Buck2 allows them (but with different meaning and different checking). -* And finally, a non-goal is to provide a complete type system capable of representing every type invariant: it's intended to be a lossy approximation. +- Reuse the existing machinery of Starlark as much as possible, avoiding + inventing a special class of type values. As a consequence, any optimisations + for values like string/list are reused. +- Provide a pleasing syntax. +- Some degree of compatibility with Python, which allows types as expressions in + the same places Buck2 allows them (but with different meaning and different + checking). +- And finally, a non-goal is to provide a complete type system capable of + representing every type invariant: it's intended to be a lossy approximation. -In addition to these built-in types, records and enumerations are provided as special concepts. +In addition to these built-in types, records and enumerations are provided as +special concepts. ## Record types @@ -64,15 +87,22 @@ For example: MyRecord = record(host=str, port=int) ``` -This above statement defines a record `MyRecord` with 2 fields, the first named `host` that must be of type `str`, and the second named `port` that must be of type `int`. +This above statement defines a record `MyRecord` with 2 fields, the first named +`host` that must be of type `str`, and the second named `port` that must be of +type `int`. Now `MyRecord` is defined, it's possible to do the following: -* Create values of this type with `MyRecord(host="localhost", port=80)`. It is a runtime error if any arguments are missed, of the wrong type, or if any unexpected arguments are given. -* Get the type of the record suitable for a type annotation with `MyRecord`. -* Get the fields of the record. For example, `v = MyRecord(host="localhost", port=80)` will provide `v.host == "localhost"` and `v.port == 80`. Similarly, `dir(v) == ["host", "port"]`. +- Create values of this type with `MyRecord(host="localhost", port=80)`. It is a + runtime error if any arguments are missed, of the wrong type, or if any + unexpected arguments are given. +- Get the type of the record suitable for a type annotation with `MyRecord`. +- Get the fields of the record. For example, + `v = MyRecord(host="localhost", port=80)` will provide `v.host == "localhost"` + and `v.port == 80`. Similarly, `dir(v) == ["host", "port"]`. -It is also possible to specify default values for parameters using the `field` function. +It is also possible to specify default values for parameters using the `field` +function. For example: @@ -80,9 +110,11 @@ For example: MyRecord = record(host=str, port=field(int, 80)) ``` -Now the `port` field can be omitted, defaulting to `80` is not present (for example, `MyRecord(host="localhost").port == 80`). +Now the `port` field can be omitted, defaulting to `80` is not present (for +example, `MyRecord(host="localhost").port == 80`). -Records are stored deduplicating their field names, making them more memory efficient than dictionaries. +Records are stored deduplicating their field names, making them more memory +efficient than dictionaries. ## Enum types @@ -94,14 +126,22 @@ For example: MyEnum = enum("option1", "option2", "option3") ``` -This statement defines an enumeration `MyEnum` that consists of the three values `"option1"`, `"option2"` and `"option3"`. +This statement defines an enumeration `MyEnum` that consists of the three values +`"option1"`, `"option2"` and `"option3"`. Now `MyEnum` is defined, it's possible to do the following: -* Create values of this type with `MyEnum("option2")`. It is a runtime error if the argument is not one of the predeclared values of the enumeration. -* Get the type of the enum suitable for a type annotation with `MyEnum`. -* Given a value of the enum (for example, `v = MyEnum("option2")`), get the underlying value `v.value == "option2"` or the index in the enumeration `v.index == 1`. -* Get a list of the values that make up the array with `MyEnum.values() == ["option1", "option2", "option3"]`. -* Treat `MyEnum` a bit like an array, with `len(MyEnum) == 3`, `MyEnum[1] == MyEnum("option2")` and iteration over enums `[x.value for x in MyEnum] == ["option1", "option2", "option3"]`. - -Enumeration types store each value once, which are then efficiently referenced by enumeration values. +- Create values of this type with `MyEnum("option2")`. It is a runtime error if + the argument is not one of the predeclared values of the enumeration. +- Get the type of the enum suitable for a type annotation with `MyEnum`. +- Given a value of the enum (for example, `v = MyEnum("option2")`), get the + underlying value `v.value == "option2"` or the index in the enumeration + `v.index == 1`. +- Get a list of the values that make up the array with + `MyEnum.values() == ["option1", "option2", "option3"]`. +- Treat `MyEnum` a bit like an array, with `len(MyEnum) == 3`, + `MyEnum[1] == MyEnum("option2")` and iteration over enums + `[x.value for x in MyEnum] == ["option1", "option2", "option3"]`. + +Enumeration types store each value once, which are then efficiently referenced +by enumeration values. diff --git a/docs/values.md b/docs/values.md index 7938faa1e..6bc7ae0a5 100644 --- a/docs/values.md +++ b/docs/values.md @@ -1,21 +1,29 @@ # Value Representation -:::warning -Some of the information in this page is outdated. However, the explanation of the problem, and thought process behind it, remains useful. Of particular note is that a garbage collected heap is now used for `Value`. -::: +:::warning Some of the information in this page is outdated. However, the +explanation of the problem, and thought process behind it, remains useful. Of +particular note is that a garbage collected heap is now used for `Value`. ::: -This page explains how values are represented in the Starlark interpreter, ignoring some incidental details. +This page explains how values are represented in the Starlark interpreter, +ignoring some incidental details. -Importantly, in Starlark, any identifiers from modules that you import are 'frozen', which means that, if you have a module that defines a list, then once you have imported the module, the list is now immutable. This design means that you can safely share imports with multiple users, without any expensive copying, and use the imports in parallel. +Importantly, in Starlark, any identifiers from modules that you import are +'frozen', which means that, if you have a module that defines a list, then once +you have imported the module, the list is now immutable. This design means that +you can safely share imports with multiple users, without any expensive copying, +and use the imports in parallel. ## Frozen vs unfrozen values Values that are frozen are segregated from those that are not: -* Frozen values are those you import, and (assuming no GC) are to be ref-counted atomically (so they can be shared by multiple threads) and never changed. -* Unfrozen values are those which are local to the module, and, since modules execute single threaded, can be non-atomically ref-counted and mutated. +- Frozen values are those you import, and (assuming no GC) are to be ref-counted + atomically (so they can be shared by multiple threads) and never changed. +- Unfrozen values are those which are local to the module, and, since modules + execute single threaded, can be non-atomically ref-counted and mutated. -Once a module has finished executing, it's values are frozen and can be reused freely. +Once a module has finished executing, it's values are frozen and can be reused +freely. ## Thaw-on-write @@ -28,11 +36,19 @@ def my_list(x): return ([1,2,3], x) ``` -This above code returns the unfrozen list `[1,2,3]`. But while the list is unfrozen, and could be mutated by the caller, it probably won't be. To optimise this pattern, construct a frozen list when compiling `my_list` and insert a shared reference to it in the result. If anyone tries to mutate the list, it's explicitly unfrozen by copying it into a mutable variant (known as thawing the value). +This above code returns the unfrozen list `[1,2,3]`. But while the list is +unfrozen, and could be mutated by the caller, it probably won't be. To optimise +this pattern, construct a frozen list when compiling `my_list` and insert a +shared reference to it in the result. If anyone tries to mutate the list, it's +explicitly unfrozen by copying it into a mutable variant (known as thawing the +value). ## Immutable containers of mutable data -There are some data types (such as functions and tuples) that are themselves immutable but contain mutable data. Importantly, all types that can be invoked as functions (for example, `lambda`, `def`, and `a.b()`) fall into this category. These types can be non-atomically ref-counted but can't be mutated. +There are some data types (such as functions and tuples) that are themselves +immutable but contain mutable data. Importantly, all types that can be invoked +as functions (for example, `lambda`, `def`, and `a.b()`) fall into this +category. These types can be non-atomically ref-counted but can't be mutated. ## Implementation in Rust @@ -58,16 +74,18 @@ enum Mutable { } ``` -In the above code, both of the traits `dyn SimpleValue` `and dyn ComplexValue` enable you to convert to the other and have shared general value-like methods. +In the above code, both of the traits `dyn SimpleValue` `and dyn ComplexValue` +enable you to convert to the other and have shared general value-like methods. There are four types of value: -* `Immutable` -* `Pseudo` - immutable containers of mutable values. -* `Mutable`/`Mutable` -* `Mutable`/`ThawOnWrite` - immutable now but can be replaced with `Mutable`/`Mutable` if needed. +- `Immutable` +- `Pseudo` - immutable containers of mutable values. +- `Mutable`/`Mutable` +- `Mutable`/`ThawOnWrite` - immutable now but can be replaced with + `Mutable`/`Mutable` if needed. There are two root types: -* `FrozenValue` - imported. -* `Value` - defined locally. +- `FrozenValue` - imported. +- `Value` - defined locally. diff --git a/gazebo/README.md b/gazebo/README.md index a41cf04d5..23da12907 100644 --- a/gazebo/README.md +++ b/gazebo/README.md @@ -6,41 +6,75 @@ [![docs.rs availability](https://img.shields.io/docsrs/gazebo?label=docs.rs)](https://docs.rs/gazebo/) [![Build status](https://img.shields.io/github/workflow/status/facebookincubator/gazebo/ci.svg)](https://github.com/facebookincubator/gazebo/actions) -This library contains a collection of well-tested utilities. Most modules stand alone, but taking a few representative examples: +This library contains a collection of well-tested utilities. Most modules stand +alone, but taking a few representative examples: -* `gazebo::prelude::*` is intended to be imported as such, and provides extension traits to common types. For example, it provides `Vec::map` which is equivalent to `iter().map(f).collect::>()`, and `str::split1` like `split` but which only splits once. We hope some of these functions one day make it into the Rust standard library. -* `gazebo::dupe` provides the trait `Dupe` with the member `dupe`, all of which are exactly like `Clone`. The difference is that `Dupe` should not be implemented for types that reallocate or have expensive `clone` operations - e.g. there is `Dupe` for `Arc` and `usize`, but not for `String` and `Vec`. By using `dupe` it is easy to focus on the `clone` calls (which should be rare) and ignore things whose cost is minimal. -* `gazebo::cell::ARef` provides a type which is either a `Ref` or a direct reference `&T`, with operations that make it look like `Ref` -- allowing you to uniformly convert a reference into something like a `Ref`. +- `gazebo::prelude::*` is intended to be imported as such, and provides + extension traits to common types. For example, it provides `Vec::map` which is + equivalent to `iter().map(f).collect::>()`, and `str::split1` like + `split` but which only splits once. We hope some of these functions one day + make it into the Rust standard library. +- `gazebo::dupe` provides the trait `Dupe` with the member `dupe`, all of which + are exactly like `Clone`. The difference is that `Dupe` should not be + implemented for types that reallocate or have expensive `clone` operations - + e.g. there is `Dupe` for `Arc` and `usize`, but not for `String` and `Vec`. By + using `dupe` it is easy to focus on the `clone` calls (which should be rare) + and ignore things whose cost is minimal. +- `gazebo::cell::ARef` provides a type which is either a `Ref` or a direct + reference `&T`, with operations that make it look like `Ref` -- allowing you + to uniformly convert a reference into something like a `Ref`. -The functionality provided by Gazebo is not stable, and continues to evolve with both additions (as we find new useful features) and removals (as we find better patterns or libraries encapsulating the ideas better). While the code varies in usefulness and design quality, it is all well tested and documented. +The functionality provided by Gazebo is not stable, and continues to evolve with +both additions (as we find new useful features) and removals (as we find better +patterns or libraries encapsulating the ideas better). While the code varies in +usefulness and design quality, it is all well tested and documented. ## Using Gazebo -Gazebo can be depended upon by adding `gazebo` to your `[dependencies]`, using the standard [Cargo patterns](https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html). +Gazebo can be depended upon by adding `gazebo` to your `[dependencies]`, using +the standard +[Cargo patterns](https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html). -The two interesting directories in this repo are `gazebo` (which contains the source to Gazebo itself) and `gazebo_derive` (which contains support for `#[derive(Dupe)]` and other Gazebo traits). Usually you will directly import `gazebo`, but `gazebo_derive` is a required transitive dependency if you are sourcing the library from GitHub. +The two interesting directories in this repo are `gazebo` (which contains the +source to Gazebo itself) and `gazebo_derive` (which contains support for +`#[derive(Dupe)]` and other Gazebo traits). Usually you will directly import +`gazebo`, but `gazebo_derive` is a required transitive dependency if you are +sourcing the library from GitHub. ## Learn More -You can learn more about Gazebo in [this introductory video](https://www.youtube.com/watch?v=pQJkx9HL_04), or from the following blog posts: +You can learn more about Gazebo in +[this introductory video](https://www.youtube.com/watch?v=pQJkx9HL_04), or from +the following blog posts: -* [Rust Nibbles - Gazebo: Prelude](https://developers.facebook.com/blog/post/2021/06/29/rust-nibbles-gazebo-prelude/) -* [Rust Nibbles - Gazebo: Dupe](https://developers.facebook.com/blog/post/2021/07/06/rust-nibbles-gazebo-dupe/) -* [Rust Nibbles - Gazebo: Variants](https://developers.facebook.com/blog/post/2021/07/13/rust-nibbles-gazebo-variants) -* [Rust Nibbles - Gazebo: AnyLifetime](https://developers.facebook.com/blog/post/2021/07/20/rust-nibbles-gazebo-any-lifetime/) -* [Rust Nibbles - Gazebo: Comparisons](https://developers.facebook.com/blog/post/2021/07/27/rust-nibbles-gazebo-comparisons/) -* [Rust Nibbles - Gazebo: Casts and Transmute](https://developers.facebook.com/blog/post/2021/08/03/rust-nibbles-gazebo-casts-transmute/) -* [Rust Nibbles - Gazebo: The rest of the tent](https://developers.facebook.com/blog/post/2021/08/10/rust-nibbles-gazebo-rest-of-tent/) +- [Rust Nibbles - Gazebo: Prelude](https://developers.facebook.com/blog/post/2021/06/29/rust-nibbles-gazebo-prelude/) +- [Rust Nibbles - Gazebo: Dupe](https://developers.facebook.com/blog/post/2021/07/06/rust-nibbles-gazebo-dupe/) +- [Rust Nibbles - Gazebo: Variants](https://developers.facebook.com/blog/post/2021/07/13/rust-nibbles-gazebo-variants) +- [Rust Nibbles - Gazebo: AnyLifetime](https://developers.facebook.com/blog/post/2021/07/20/rust-nibbles-gazebo-any-lifetime/) +- [Rust Nibbles - Gazebo: Comparisons](https://developers.facebook.com/blog/post/2021/07/27/rust-nibbles-gazebo-comparisons/) +- [Rust Nibbles - Gazebo: Casts and Transmute](https://developers.facebook.com/blog/post/2021/08/03/rust-nibbles-gazebo-casts-transmute/) +- [Rust Nibbles - Gazebo: The rest of the tent](https://developers.facebook.com/blog/post/2021/08/10/rust-nibbles-gazebo-rest-of-tent/) ## Making a release -1. Check the [GitHub Actions](https://github.com/facebookincubator/gazebo/actions) are green. -2. Update `CHANGELOG.md` with the changes since the last release. [This link](https://github.com/facebookincubator/gazebo/compare/v0.1.0...main) can help (update to compare against the last release). -3. Update the version numbers of the two `Cargo.toml` files. Bump them by 0.0.1 if there are no incompatible changes, or 0.1.0 if there are. Bump the dependency in `gazebo` to point at the latest `gazebo_derive` version. -4. Copy the files `CHANGELOG.md`, the two `LICENSE-` files and `README.md` into each `gazebo` and `gazebo_derive` subdirectory. -5. Run `cargo publish --allow-dirty --dry-run`, then without the `--dry-run`, first in `gazebo_derive` and then `gazebo` directories. -6. Create a [GitHub release](https://github.com/facebookincubator/gazebo/releases/new) with `v0.X.Y`, using the `gazebo` version as the name. +1. Check the + [GitHub Actions](https://github.com/facebookincubator/gazebo/actions) are + green. +2. Update `CHANGELOG.md` with the changes since the last release. + [This link](https://github.com/facebookincubator/gazebo/compare/v0.1.0...main) + can help (update to compare against the last release). +3. Update the version numbers of the two `Cargo.toml` files. Bump them by 0.0.1 + if there are no incompatible changes, or 0.1.0 if there are. Bump the + dependency in `gazebo` to point at the latest `gazebo_derive` version. +4. Copy the files `CHANGELOG.md`, the two `LICENSE-` files and `README.md` into + each `gazebo` and `gazebo_derive` subdirectory. +5. Run `cargo publish --allow-dirty --dry-run`, then without the `--dry-run`, + first in `gazebo_derive` and then `gazebo` directories. +6. Create a + [GitHub release](https://github.com/facebookincubator/gazebo/releases/new) + with `v0.X.Y`, using the `gazebo` version as the name. ## License -Gazebo is both MIT and Apache License, Version 2.0 licensed, as found in the [LICENSE-MIT](LICENSE-MIT) and [LICENSE-APACHE](LICENSE-APACHE) files. +Gazebo is both MIT and Apache License, Version 2.0 licensed, as found in the +[LICENSE-MIT](LICENSE-MIT) and [LICENSE-APACHE](LICENSE-APACHE) files. diff --git a/starlark/testcases/eval/go/README.md b/starlark/testcases/eval/go/README.md index 4eab164d7..f078fc4f3 100644 --- a/starlark/testcases/eval/go/README.md +++ b/starlark/testcases/eval/go/README.md @@ -1,5 +1,7 @@ # Go evaluation test cases -The Go Starlark project maintains a set of test cases, which were mirrored here. The original source -is https://github.com/google/starlark-go/blob/e81fc95f7bd5bb1495fe69f27c1a99fcc77caa48/starlark/testdata/. -Note that some files were not copied, because they are unsuitable tests for Starlark, as described in the `test_go` function. +The Go Starlark project maintains a set of test cases, which were mirrored here. +The original source is +https://github.com/google/starlark-go/blob/e81fc95f7bd5bb1495fe69f27c1a99fcc77caa48/starlark/testdata/. +Note that some files were not copied, because they are unsuitable tests for +Starlark, as described in the `test_go` function. diff --git a/starlark_syntax/testcases/parse/README.md b/starlark_syntax/testcases/parse/README.md index 32c0b3925..7f59d8c1b 100644 --- a/starlark_syntax/testcases/parse/README.md +++ b/starlark_syntax/testcases/parse/README.md @@ -1,6 +1,7 @@ # Parsing test cases -A set of `.bzl` files taken from various open-source projects to test the Starlark parser against real world cases. +A set of `.bzl` files taken from various open-source projects to test the +Starlark parser against real world cases. -Files are marked generated, that was done to mute linters. There's -no automation to regenerate these files. +Files are marked generated, that was done to mute linters. There's no automation +to regenerate these files. diff --git a/vscode/README.md b/vscode/README.md index c665bbf34..06714851f 100644 --- a/vscode/README.md +++ b/vscode/README.md @@ -1,18 +1,27 @@ # Starlark VS Code LSP extension -A VSCode LSP extension that talks over stdin/stdout to a binary. This can either be the starlark binary itself, or any binary that has implemented `starlark::lsp::server::LspContext` and runs `starlark::lsp::server::stdio_server()`. +A VSCode LSP extension that talks over stdin/stdout to a binary. This can either +be the starlark binary itself, or any binary that has implemented +`starlark::lsp::server::LspContext` and runs +`starlark::lsp::server::stdio_server()`. -If using another binary, the settings to be aware of are `starlark.lspPath` (the binary path) and `starlark.lspArguments` (the arguments to that binary). These are available in the VSCode extension settings UI. +If using another binary, the settings to be aware of are `starlark.lspPath` (the +binary path) and `starlark.lspArguments` (the arguments to that binary). These +are available in the VSCode extension settings UI. Based on a combination of: -* Tutorial at https://code.visualstudio.com/api/language-extensions/language-server-extension-guide -* Code for the tutorial at https://github.com/microsoft/vscode-extension-samples/tree/master/lsp-sample -* Syntax files from https://github.com/phgn0/vscode-starlark (which are the Microsoft Python ones with minor tweaks) +- Tutorial at + https://code.visualstudio.com/api/language-extensions/language-server-extension-guide +- Code for the tutorial at + https://github.com/microsoft/vscode-extension-samples/tree/master/lsp-sample +- Syntax files from https://github.com/phgn0/vscode-starlark (which are the + Microsoft Python ones with minor tweaks) ## Pre-requisites -You need to have npm v7+ installed. Afterwards, run `npm install` in this folder and in `client`. +You need to have npm v7+ installed. Afterwards, run `npm install` in this folder +and in `client`. ## Debugging @@ -28,16 +37,22 @@ You need to have npm v7+ installed. Afterwards, run `npm install` in this folder - Follow steps in Pre-requisites section. - Run `npm install vsce` - Run `npm exec vsce package` -- In VS Code, go to Extensions, click on the "..." button in the Extensions bar, select "Install from VSIX" and then select the `starlark-1.0.0.vsix` file that was produced. -- Build the starlark binary with `cargo build --bin=starlark` and then do one of: - - Put it on your `$PATH`, e.g. `cp $CARGO_TARGET_DIR/debug/starlark ~/.cargo/bin/starlark`. - - Configure the setting `starlark.lspPath` for this extension to point to the starlark binary. e.g. `$CARGO_TARGET_DIR/debug/starlark`. +- In VS Code, go to Extensions, click on the "..." button in the Extensions bar, + select "Install from VSIX" and then select the `starlark-1.0.0.vsix` file that + was produced. +- Build the starlark binary with `cargo build --bin=starlark` and then do one + of: + - Put it on your `$PATH`, e.g. + `cp $CARGO_TARGET_DIR/debug/starlark ~/.cargo/bin/starlark`. + - Configure the setting `starlark.lspPath` for this extension to point to the + starlark binary. e.g. `$CARGO_TARGET_DIR/debug/starlark`. ## Updating -Every few months security advisories will arrive about pinned versions of packages. +Every few months security advisories will arrive about pinned versions of +packages. -* `npm audit` to see which packages have security updates. -* `npm audit fix` to fix those issues. -* Try `npm audit`, if it still has issues run `npm update`. -* `npm exec vsce package` to confirm everything still works. +- `npm audit` to see which packages have security updates. +- `npm audit fix` to fix those issues. +- Try `npm audit`, if it still has issues run `npm update`. +- `npm exec vsce package` to confirm everything still works.