Store diagnostics in memdb table

[dbanck]

This is part of hashicorp/vscode-terraform#635

Problem Statement

Currently we're sourcing and publishing diagnostics in two different places:

As result of parsing files

terraform-ls/internal/langserver/handlers/hooks_module.go

Lines 144 to 152 in f246688

	diags := diagnostics.NewDiagnostics()
	diags.EmptyRootDiagnostic()
	defer notifier.PublishHCLDiags(ctx, newMod.Path, diags)
	if newMod != nil {
	diags.Append("HCL", newMod.ModuleDiagnostics.AsMap())
	diags.Append("HCL", newMod.VarsDiagnostics.AutoloadedOnly().AsMap())
	}

terraform validate

terraform-ls/internal/langserver/handlers/command/validate.go

Lines 67 to 72 in f246688

	diags := diagnostics.NewDiagnostics()
	validateDiags := diagnostics.HCLDiagsFromJSON(jsonDiags)
	diags.EmptyRootDiagnostic()
	diags.Append("terraform validate", validateDiags)
	diags.Append("HCL", mod.ModuleDiagnostics.AsMap())
	diags.Append("HCL", mod.VarsDiagnostics.AutoloadedOnly().AsMap())

This can lead to stale or suddenly disappearing diagnostics, for example when validate is executed as an editor command.

Requirements Summary

• Central storage for diagnostics
• Different sources feed diagnostics for a single or multiple files
• Publishing diagnostics will always return all available diagnostics
• Diagnostics need to be refreshed, independent of each other, and not impact or clear other sources

Proposal

• Create a new memdb table for diagnostics based on the following struct

type Diagnostic struct {
   Dir      string
   Filename string
   Range    hcl.Range
   Severity hcl.DiagnosticSeverity
 
   // The diagnostic's code, which might appear in the user interface
   // e.g. "terraform_deprecated_interpolation"
   Code            string
   // An optional property to describe the error code
   CodeDescription string
 
   // A human-readable string describing the source, e.g. "tflint"
   Source  string
   // The diagnostic's message.
   // e.g. "Interpolation-only expressions are deprecated"
   Message string
 
   // Optional data entry field
   Data interface{}
}

• Store HCL diagnostics in the table
• Store validate diagnostics in the table
• Query diagnostics table when publishing diagnostics

Related LSP methods

• textDocument/publishDiagnostics

Upcoming diagnostics sources

• Provide diagnostics and quick fixes for invalid configuration vscode-terraform#720
• Integrate tflint with focus on a generalized linter/validator interface #334

Open Questions

• Could memdb "watchers" be used to publish diagnostics?

[radeksimko]

@jpogran & @dbanck will take a look if we could avoid storing diagnostics in memdb entirely

James expects the LSP spec to differ from the reality in VS Code and/or other clients.

[radeksimko]

I have not looked into gopls codebase for the implementation and current state, but at least last year in 2021 they stored diagnostics per files according to this microsoft/language-server-protocol#1217

[dbanck]

It's no longer needed due to #1667 and the diagnostic source map we introduced as part of the enhanced validation work.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.