EL9 RPM Sqlite3 DB support #201

agrare · 2024-10-31T14:55:17Z

Depends on:

Fryguy · 2024-10-31T14:57:24Z

I'd be happy merging specs as is ahead of actually implementing anything.

agrare · 2024-10-31T14:59:30Z

spec/metadata/linux/LinuxPackages_spec.rb

+      it "returns a list of rpm packages" do
+        result = described_class.new(fs)
+
+        expect(result.instance_variable_get(:@packages).count).to eq(690)


I'm not sure why we don't have an attr_reader for @packages here, toXml and toString are the only "interface" methods but I think access to packages would be handy also

Fryguy · 2024-10-31T18:42:47Z

lib/metadata/linux/MiqRpmPackages/Bdb.rb

+    }
+
+    #
+    # Nubbers on disk are in network byte order.


LOL nubbers

Fryguy · 2024-11-04T17:31:00Z

lib/db/MiqSqlite/MiqSqlite3Table.rb

@@ -63,7 +63,7 @@ def decodeSchema
      return if @type != "table" || @name[0..6] == "sqlite_"  # Names beginning with sqlite_ are internal to engine
      @columns = []
      sql = @sql.gsub(/[\n\r]/, "")
-      re1 = /\s*CREATE\s+TABLE\s+(\w+)\s*\((.*)\)\s*/
+      re1 = /\s*CREATE\s+TABLE\s+'?(\w+)'?\s*\((.*)\)\s*/


That's interesting - I've never seen quotes before. Also note that technically the table name could have a . in it as well (i.e. <schema_name>.<table_name>)

Fryguy · 2024-11-04T17:32:17Z

@agrare More out of curiosity, what's the approach here? I'm seeing refactoring into separate modules + specs...is the plan to do an initial refactor, then merge, then add the new DB support?

agrare · 2024-11-04T17:37:33Z

@Fryguy so I see this as two high level issues,

MiqSqlite3DB can't currently load the rpmdb.sqlite database
MiqRpmPackages can't currently process a rpmdb.sqlite file

So the plan is:

Get MiqSqlite3DB::Sqlite3 able to load the rpmdb.sqlite file and list tables
Get MiqSqlite3DB::Sqlite3Table able to list rows in the Packages table (this is also broken currently)
Get MiqLinux::MiqRpmPackages able to understand the contents of the rpmdb.sqlite database (this PR)

So I'm rebasing this on top of the sqlite3 fixes

Fryguy · 2024-11-04T17:40:13Z

Moving forward I wonder if it's better to use the sqlite library instead of rolling our own. Not sure if that is with investigation or not. I'd rather do whats more expedient at this time, but I'm also wondering what bugs will lurk in the corners.

agrare · 2024-11-04T17:41:29Z

Moving forward I wonder if it's better to use the sqlite library instead of rolling our own

Yes I did look into that originally, but I don't think that will work with the way that we handle loading the filesystem via MiqFs in SSA. We'd need a "real" file since the interface is a path and all of the loading of the database is done in C-ext code.

(Another reason I'd like to use FUSE instead eventually so that we could "mount" the filesystem and use more standardized tools based on files/paths)

lib/metadata/linux/MiqRpmPackages/Sqlite.rb

Fryguy · 2025-01-16T19:16:49Z

Nice - tests went green.

agrare · 2025-01-16T20:36:46Z

Okay checklist of things that we need for this to be merged:

ruby-rpm-ffi support for rpm 4.14+ (mostly working in master, but no release cut) so that requiring the gem doesn't blow up with librpm.so.8+
ruby-rpm-ffi add detection of librpm.so.9 and librpm.so.10 (we can work around this by installing rpm-devel so ruby-rpm-ffi picks up on librpm.so but that is sub-optimal)

agrare · 2025-02-03T17:36:16Z

spec/metadata/linux/data/rpm/rpmdb.sqlite

@Fryguy I took the rpmdb.sqlite from one of the ruby-rpm-ffi tests where it installs a single dummy package so that we can actually test the package parser code above instead of just expect(result.packages.count).to eq(0)

agrare · 2025-02-03T20:48:03Z

I've updated this to use the forked gem, trying to figure out why this is red on CI but green locally

agrare · 2025-02-04T17:44:47Z

Okay taking out of WIP as this is passing specs with a populated rpmdb.sqlite database, we can figure out the behavior of the _dbpath on mac now or in a follow-up

Fryguy · 2025-02-04T17:56:39Z

lib/metadata/linux/LinuxPackages.rb

-      rpmp.each { |p| @packages << p }
-      rpmp.close
+
+      rpmp = MiqRpmPackages.new(@fs, dbDir)


Eventually we should add a block form of this that auto-closes

Improve the specs by testing with an rpmdb.sqlite that has a package installed

agrare · 2025-02-04T18:21:12Z

NOTE in order to support EL10 we'll need to check /var/lib/rpm as well as /usr/lib/sysimage/rpm

Fryguy · 2025-02-04T19:18:34Z

NOTE in order to support EL10 we'll need to check /var/lib/rpm as well as /usr/lib/sysimage/rpm

would that be covered by _dbpath?

Fryguy · 2025-02-04T19:21:14Z

lib/metadata/linux/MiqRpmPackages.rb

-      s = getStringVal(data, cpos)
-      ra << s
-      cpos += (s.length + 1)
+      orig_new(fs, dbDir)


I think we can use a super pattern here instead of the orig_new

Even so, I will merge without this.

I pulled this pattern from linux_admin, if this isn't the best way to do it we should probably change both then

agrare · 2025-02-04T19:21:52Z

RPM["_dbpath"] covers where the local system thinks the db should be, when we look to the system that we're scanning the db path could be one or the other depending on the version of that system.

Added * Support scanning RPM packages on EL8/EL9 (#201) * Test with ruby 3.3 (#208)

Fryguy self-assigned this Oct 31, 2024

agrare commented Oct 31, 2024

View reviewed changes

agrare force-pushed the miq_rpm_packages_el9 branch from c5441ac to 6cff95a Compare October 31, 2024 15:11

agrare mentioned this pull request Oct 31, 2024

Add specs for MiqLinux::Packages for rpm/dpkg #202

Merged

agrare force-pushed the miq_rpm_packages_el9 branch from 6cff95a to 43f8f90 Compare October 31, 2024 17:47

Fryguy reviewed Oct 31, 2024

View reviewed changes

lib/metadata/linux/MiqRpmPackages/Bdb.rb

}

#

# Nubbers on disk are in network byte order.

Copy link

Member

Fryguy Oct 31, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LOL nubbers

agrare force-pushed the miq_rpm_packages_el9 branch 2 times, most recently from cd7f0b9 to 654c48a Compare November 4, 2024 16:35

Fryguy reviewed Nov 4, 2024

View reviewed changes

agrare force-pushed the miq_rpm_packages_el9 branch 2 times, most recently from 6adc493 to 37ca67f Compare November 4, 2024 18:05

agrare mentioned this pull request Nov 5, 2024

[RFE] Support RHEL / CentOS Stream 9 #180

Open

4 tasks

agrare force-pushed the miq_rpm_packages_el9 branch from 37ca67f to 500fdc0 Compare January 9, 2025 20:58

Fryguy reviewed Jan 16, 2025

View reviewed changes

lib/metadata/linux/MiqRpmPackages/Sqlite.rb Outdated Show resolved Hide resolved

Fryguy reviewed Jan 16, 2025

View reviewed changes

lib/metadata/linux/MiqRpmPackages/Sqlite.rb Outdated Show resolved Hide resolved

agrare added the enhancement label Jan 16, 2025

agrare force-pushed the miq_rpm_packages_el9 branch from eaed4dd to 001598b Compare February 3, 2025 17:34

agrare commented Feb 3, 2025

View reviewed changes

agrare force-pushed the miq_rpm_packages_el9 branch 3 times, most recently from 42c42bf to 05b2b28 Compare February 4, 2025 16:28

agrare mentioned this pull request Feb 4, 2025

Add rpmdb.sqlite with packages #212

Closed

1 task

agrare force-pushed the miq_rpm_packages_el9 branch from 4f594eb to acd4bb3 Compare February 4, 2025 17:53

Fryguy reviewed Feb 4, 2025

View reviewed changes

agrare force-pushed the miq_rpm_packages_el9 branch from b0ef951 to 9917ffc Compare February 4, 2025 18:00

agrare mentioned this pull request Feb 4, 2025

Add MiqLinux::Packages#packages accessor #213

Merged

agrare force-pushed the miq_rpm_packages_el9 branch 2 times, most recently from 8e9df24 to 3db66ef Compare February 4, 2025 18:10

agrare and others added 13 commits February 4, 2025 13:11

Split MiqRpmPackages based on backing database

71646cd

Extract rpmdb.sqlite from MiqFs

4ea9a51

Use RPM FFI gem to parse rpmdb.sqlite

1debc41

Yield a compatible hash struct from Sqlite/Bdb

1b07c5d

Ensure that the rpmdb is closed

5061e88

Add support for sqlite smartstate from a macOS host

2900afa

Fix issue where on Linux we were using an absolute path

5d53777

Install librpm-dev for RPM specs

39d9484

Replace fs.present? with fs.nil?

081da05

Switch to rpm2 gem

ffa1e41

Use rpmdb.sqlite with a single sample package

9904680

Improve the specs by testing with an rpmdb.sqlite that has a package installed

Use RPM _dbpath constant

e4c1ff1

Fix circular require in MiqRpmPackages::Sqlite

e261933

agrare force-pushed the miq_rpm_packages_el9 branch from 3db66ef to e261933 Compare February 4, 2025 18:12

Fryguy reviewed Feb 4, 2025

View reviewed changes

Fryguy approved these changes Feb 4, 2025

View reviewed changes

Fryguy merged commit 67de160 into ManageIQ:master Feb 4, 2025
3 of 4 checks passed

agrare deleted the miq_rpm_packages_el9 branch February 4, 2025 19:25

agrare added a commit that referenced this pull request Feb 4, 2025

Release v0.11.0

f1f4d2a

Added * Support scanning RPM packages on EL8/EL9 (#201) * Test with ruby 3.3 (#208)

agrare mentioned this pull request Feb 4, 2025

Bump manageiq-smartstate for EL8,9 RPM support ManageIQ/manageiq#23326

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

EL9 RPM Sqlite3 DB support #201

EL9 RPM Sqlite3 DB support #201

agrare commented Oct 31, 2024 •

edited by Fryguy

Loading

Fryguy commented Oct 31, 2024

agrare Oct 31, 2024

Fryguy Oct 31, 2024

Fryguy Nov 4, 2024

Fryguy commented Nov 4, 2024

agrare commented Nov 4, 2024 •

edited

Loading

Fryguy commented Nov 4, 2024

agrare commented Nov 4, 2024 •

edited

Loading

Fryguy commented Jan 16, 2025

agrare commented Jan 16, 2025 •

edited

Loading

agrare Feb 3, 2025

agrare commented Feb 3, 2025

agrare commented Feb 4, 2025

Fryguy Feb 4, 2025

agrare commented Feb 4, 2025

Fryguy commented Feb 4, 2025

Fryguy Feb 4, 2025

Fryguy Feb 4, 2025

agrare Feb 4, 2025

agrare commented Feb 4, 2025

EL9 RPM Sqlite3 DB support #201

EL9 RPM Sqlite3 DB support #201

Conversation

agrare commented Oct 31, 2024 • edited by Fryguy Loading

Fryguy commented Oct 31, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Fryguy commented Nov 4, 2024

agrare commented Nov 4, 2024 • edited Loading

Fryguy commented Nov 4, 2024

agrare commented Nov 4, 2024 • edited Loading

Fryguy commented Jan 16, 2025

agrare commented Jan 16, 2025 • edited Loading

Choose a reason for hiding this comment

agrare commented Feb 3, 2025

agrare commented Feb 4, 2025

Choose a reason for hiding this comment

agrare commented Feb 4, 2025

Fryguy commented Feb 4, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

agrare commented Feb 4, 2025

agrare commented Oct 31, 2024 •

edited by Fryguy

Loading

agrare commented Nov 4, 2024 •

edited

Loading

agrare commented Nov 4, 2024 •

edited

Loading

agrare commented Jan 16, 2025 •

edited

Loading