diff --git a/_data/elixir_wizards_episodes.yml b/_data/elixir_wizards_episodes.yml
index c327fdd5..d95602d4 100644
--- a/_data/elixir_wizards_episodes.yml
+++ b/_data/elixir_wizards_episodes.yml
@@ -104,6 +104,176 @@
some of the topics we’ll be covering.\n\nLearn more about how SmartLogic
uses Phoenix and Elixir.
\n
\ "
+- title: '"Saga of a Gnarly Report" with Owen and Dan'
+ slug: s12-e05-saga-of-a-gnarly-report
+ link: https://smartlogic.io/podcast/elixir-wizards/s12-e05-saga-of-a-gnarly-report
+ guid: f0d1daa0-bc06-4daf-b6c9-c37a12357821
+ pubDate: Thu, 18 Apr 2024 07:00:00 -0400
+ pubDateFriendly: April 18, 2024
+ description: "In today's episode, Elixir Wizards Owen and Dan delve into the complexities
+ of building advanced reporting features within software applications. They share
+ personal insights and challenges encountered while developing reporting solutions
+ for user-generated data, leveraging both Elixir/Phoenix and Ruby on Rails.\nThe
+ discussion zeroes in on crucial data modeling and architectural decisions that
+ enhance reporting efficiency and flexibility. Owen and Dan explore tactics like
+ materialized views, event sourcing, and database triggers to optimize data handling
+ while being mindful of UX elements like progress indicators and background job
+ management.\nThey share insights on leveraging the Elixir/Beam ecosystem’s strengths—like
+ concurrency and streamlined deployment—to tackle common reporting, caching, and
+ integration challenges. The episode highlights the impact of reporting features
+ across all aspects of a software application’s design and architecture.\nKey topics
+ discussed in this episode:\nReporting on assessment data, survey results, and
+ user metrics\nDifferences between reporting and performance/error monitoring\nImplementing
+ reporting in Elixir/Phoenix vs. Ruby on Rails\nDisplaying reports in web, printable,
+ PDF, SVG, and CSV formats\nChallenges of generating PDFs for large data sets\nStreaming
+ CSV data directly to the client\nHandling long-running report generation tasks\nProviding
+ progress indicators and user notifications\nStrategies for canceling or abandoning
+ incomplete reports\nTradeoffs of pre-calculating report data vs. real-time generation\nMaterializing
+ views and denormalizing data for reporting\nExploring event sourcing patterns
+ for reporting needs\nUsing database triggers and stored procedures for reporting\nBalancing
+ data structure optimization for reports vs. day-to-day usage\nCaching report data
+ for faster retrieval and rendering\nCharting and visualization integration in
+ reporting systems\nLinks mentioned:\nPrometheus monitoring system & time series
+ database https://prometheus.io/ \nThinking Elixir \"FLAME with Chris McCord\"
+ https://podcast.thinkingelixir.com/181\nPhoenix LiveView Uploads https://hexdocs.pm/phoenix/fileuploads.html\nhttps://hexdocs.pm/phoenixlive_view/Phoenix.LiveView.UploadWriter.html
+ \nPostgrex PostgreSQL driver for Elixir https://hexdocs.pm/postgrex/Postgrex.html
+ \nEcto https://hexdocs.pm/ecto/Ecto.html \nHeroku cloud application platform
+ https://www.heroku.com/ \nElixir Wizards S9E12 Marcelo Dominguez on Command and
+ Query Responsibility Segregation https://smartlogic.io/podcast/elixir-wizards/s9-e12-marcelo-dominguez-cqrs/\nCommanded
+ Elixir CQRS/ES applications https://github.com/commanded/commanded\nTailwind CSS
+ Framework https://github.com/tailwindlabs\nMemcached https://memcached.org/\nRedis
+ https://redis.io/\nOban https://hexdocs.pm/oban/Oban.html\nETS https://hexdocs.pm/ets/ETS.html\nCapistrano
+ remote server automation and deployment tool https://capistranorb.com/ \n"
+ author: SmartLogic LLC
+ embedUrl: https://fireside.fm/player/v2/IAs5ixts+gRrFRMaV
+ enclosure:
+ url: https://aphid.fireside.fm/d/1437767933/03a50f66-dc5e-4da4-ab6e-31895b6d4c9e/f0d1daa0-bc06-4daf-b6c9-c37a12357821.mp3
+ length: '97997538'
+ type: audio/mpeg
+ itunes:
+ episodeType: full
+ season: '12'
+ author: SmartLogic LLC
+ subtitle: Elixir Wizards Owen and Dan delve into the complexities of building
+ advanced reporting features within software applications. They share personal
+ insights and challenges encountered while developing reporting solutions for
+ user-generated data, leveraging both Elixir/Phoenix and Ruby on Rails.
+ duration: '50:21'
+ explicit: 'no'
+ keywords: software development, web development, programming languages, software
+ engineering, data reporting, app development, user experience, Ruby on Rails,
+ Elixir programming, Phoenix framework, application design, system architecture,
+ API development, real-time applications, open-source software, software design
+ principles, user interface design, project management, tech industry trends,
+ agile methodology, database management, SQL, NoSQL, cloud computing, frontend
+ development, backend development, full-stack development, JavaScript, Python,
+ HTML, CSS, data visualization, machine learning, artificial intelligence, software
+ testing, continuous integration, deployment strategies, version control, Git,
+ GitHub, DevOps, security best practices, network architecture, data science,
+ data analysis, big data, software documentation, coding best practices, code
+ refactoring, system scalability, performance optimization, server-side scripting,
+ client-side scripting, responsive design, mobile app development, cross-platform
+ development, tech startups, tech news, digital transformation, IT management,
+ tech conferences, programming tutorials, coding challenges, software career
+ development
+ image: https://assets.fireside.fm/file/fireside-images/podcasts/images/0/03a50f66-dc5e-4da4-ab6e-31895b6d4c9e/episodes/f/f0d1daa0-bc06-4daf-b6c9-c37a12357821/cover.jpg
+ summary: "\n In today's episode, Elixir Wizards Owen and Dan delve
+ into the complexities of building advanced reporting features within software
+ applications. They share personal insights and challenges encountered while
+ developing reporting solutions for user-generated data, leveraging both Elixir/Phoenix
+ and Ruby on Rails.
\n\nThe discussion zeroes in on crucial data modeling
+ and architectural decisions that enhance reporting efficiency and flexibility.
+ Owen and Dan explore tactics like materialized views, event sourcing, and database
+ triggers to optimize data handling while being mindful of UX elements like progress
+ indicators and background job management.
\n\nThey share insights on leveraging
+ the Elixir/Beam ecosystem’s strengths—like concurrency and streamlined deployment—to
+ tackle common reporting, caching, and integration challenges. The episode highlights
+ the impact of reporting features across all aspects of a software application’s
+ design and architecture.
\n\nKey topics discussed in this episode:
\n\n\n- Reporting
+ on assessment data, survey results, and user metrics
\n- Differences between
+ reporting and performance/error monitoring
\n- Implementing reporting
+ in Elixir/Phoenix vs. Ruby on Rails
\n- Displaying reports in web, printable,
+ PDF, SVG, and CSV formats
\n- Challenges of generating PDFs for large
+ data sets
\n- Streaming CSV data directly to the client
\n- Handling
+ long-running report generation tasks
\n- Providing progress indicators
+ and user notifications
\n- Strategies for canceling or abandoning incomplete
+ reports
\n- Tradeoffs of pre-calculating report data vs. real-time generation
\n- Materializing
+ views and denormalizing data for reporting
\n- Exploring event sourcing
+ patterns for reporting needs
\n- Using database triggers and stored procedures
+ for reporting
\n- Balancing data structure optimization for reports vs.
+ day-to-day usage
\n- Caching report data for faster retrieval and rendering
\n- Charting
+ and visualization integration in reporting systems
\n
\n\nLinks
+ mentioned:
\n\nPrometheus monitoring system & time series database
+ https://prometheus.io/
+
\nThinking Elixir "FLAME with Chris McCord" https://podcast.thinkingelixir.com/181
\nPhoenix LiveView
+ Uploads https://hexdocs.pm/phoenix/file_uploads.html
\nhttps://hexdocs.pm/phoenix_live_view/Phoenix.LiveView.UploadWriter.html
+
\nPostgrex PostgreSQL driver for Elixir https://hexdocs.pm/postgrex/Postgrex.html
\nEcto https://hexdocs.pm/ecto/Ecto.html
+
\nHeroku cloud application platform https://www.heroku.com/
\nElixir Wizards S9E12 Marcelo
+ Dominguez on Command and Query Responsibility Segregation https://smartlogic.io/podcast/elixir-wizards/s9-e12-marcelo-dominguez-cqrs/
\nCommanded
+ Elixir CQRS/ES applications https://github.com/commanded/commanded
\nTailwind CSS
+ Framework https://github.com/tailwindlabs
\nMemcached
+ https://memcached.org/
\nRedis
+ https://redis.io/
\nOban
+ https://hexdocs.pm/oban/Oban.html
\nETS
+ https://hexdocs.pm/ets/ETS.html
\nCapistrano
+ remote server automation and deployment tool https://capistranorb.com/
\n "
+ contentEncoded: "\n In today's episode, Elixir Wizards Owen and Dan
+ delve into the complexities of building advanced reporting features within software
+ applications. They share personal insights and challenges encountered while developing
+ reporting solutions for user-generated data, leveraging both Elixir/Phoenix and
+ Ruby on Rails.
\n\nThe discussion zeroes in on crucial data modeling and
+ architectural decisions that enhance reporting efficiency and flexibility. Owen
+ and Dan explore tactics like materialized views, event sourcing, and database
+ triggers to optimize data handling while being mindful of UX elements like progress
+ indicators and background job management.
\n\nThey share insights on leveraging
+ the Elixir/Beam ecosystem’s strengths—like concurrency and streamlined deployment—to
+ tackle common reporting, caching, and integration challenges. The episode highlights
+ the impact of reporting features across all aspects of a software application’s
+ design and architecture.
\n\nKey topics discussed in this episode:
\n\n\n- Reporting
+ on assessment data, survey results, and user metrics
\n- Differences between
+ reporting and performance/error monitoring
\n- Implementing reporting in
+ Elixir/Phoenix vs. Ruby on Rails
\n- Displaying reports in web, printable,
+ PDF, SVG, and CSV formats
\n- Challenges of generating PDFs for large data
+ sets
\n- Streaming CSV data directly to the client
\n- Handling long-running
+ report generation tasks
\n- Providing progress indicators and user notifications
\n- Strategies
+ for canceling or abandoning incomplete reports
\n- Tradeoffs of pre-calculating
+ report data vs. real-time generation
\n- Materializing views and denormalizing
+ data for reporting
\n- Exploring event sourcing patterns for reporting needs
\n- Using
+ database triggers and stored procedures for reporting
\n- Balancing data
+ structure optimization for reports vs. day-to-day usage
\n- Caching report
+ data for faster retrieval and rendering
\n- Charting and visualization integration
+ in reporting systems
\n
\n\nLinks mentioned:
\n\nPrometheus
+ monitoring system & time series database https://prometheus.io/
\nThinking Elixir "FLAME
+ with Chris McCord" https://podcast.thinkingelixir.com/181
\nPhoenix
+ LiveView Uploads https://hexdocs.pm/phoenix/file_uploads.html
\nhttps://hexdocs.pm/phoenix_live_view/Phoenix.LiveView.UploadWriter.html
+
\nPostgrex PostgreSQL driver for Elixir https://hexdocs.pm/postgrex/Postgrex.html
\nEcto https://hexdocs.pm/ecto/Ecto.html
\nHeroku cloud application
+ platform https://www.heroku.com/
+
\nElixir Wizards S9E12 Marcelo Dominguez on Command and Query Responsibility
+ Segregation https://smartlogic.io/podcast/elixir-wizards/s9-e12-marcelo-dominguez-cqrs/
\nCommanded
+ Elixir CQRS/ES applications https://github.com/commanded/commanded
\nTailwind CSS
+ Framework https://github.com/tailwindlabs
\nMemcached
+ https://memcached.org/
\nRedis
+ https://redis.io/
\nOban
+ https://hexdocs.pm/oban/Oban.html
\nETS
+ https://hexdocs.pm/ets/ETS.html
\nCapistrano
+ remote server automation and deployment tool https://capistranorb.com/
\n "
- title: '"Whose Tailwind is it Anyway?" with Ava Slivkoff'
slug: s12-e04-software-design-development-collab
link: https://smartlogic.io/podcast/elixir-wizards/s12-e04-software-design-development-collab
diff --git a/_data/elixir_wizards_feed.yml b/_data/elixir_wizards_feed.yml
index 4f5ab23b..47d006bb 100644
--- a/_data/elixir_wizards_feed.yml
+++ b/_data/elixir_wizards_feed.yml
@@ -1,6 +1,6 @@
---
title: Elixir Wizards
-pubDate: Thu, 11 Apr 2024 12:30:04 -0000
+pubDate: Thu, 18 Apr 2024 11:00:04 -0000
link: https://smartlogic.io/podcast/elixir-wizards
description: "Elixir Wizards is an interview-style podcast for anyone interested in
functional programming and the Elixir Programming Language. Hosted by SmartLogic
@@ -119,6 +119,176 @@ items:
some of the topics we’ll be covering.\n\nLearn more about how SmartLogic
uses Phoenix and Elixir.
\n
\ "
+- title: '"Saga of a Gnarly Report" with Owen and Dan'
+ slug: s12-e05-saga-of-a-gnarly-report
+ link: https://smartlogic.io/podcast/elixir-wizards/s12-e05-saga-of-a-gnarly-report
+ guid: f0d1daa0-bc06-4daf-b6c9-c37a12357821
+ pubDate: Thu, 18 Apr 2024 07:00:00 -0400
+ pubDateFriendly: April 18, 2024
+ description: "In today's episode, Elixir Wizards Owen and Dan delve into the complexities
+ of building advanced reporting features within software applications. They share
+ personal insights and challenges encountered while developing reporting solutions
+ for user-generated data, leveraging both Elixir/Phoenix and Ruby on Rails.\nThe
+ discussion zeroes in on crucial data modeling and architectural decisions that
+ enhance reporting efficiency and flexibility. Owen and Dan explore tactics like
+ materialized views, event sourcing, and database triggers to optimize data handling
+ while being mindful of UX elements like progress indicators and background job
+ management.\nThey share insights on leveraging the Elixir/Beam ecosystem’s strengths—like
+ concurrency and streamlined deployment—to tackle common reporting, caching, and
+ integration challenges. The episode highlights the impact of reporting features
+ across all aspects of a software application’s design and architecture.\nKey topics
+ discussed in this episode:\nReporting on assessment data, survey results, and
+ user metrics\nDifferences between reporting and performance/error monitoring\nImplementing
+ reporting in Elixir/Phoenix vs. Ruby on Rails\nDisplaying reports in web, printable,
+ PDF, SVG, and CSV formats\nChallenges of generating PDFs for large data sets\nStreaming
+ CSV data directly to the client\nHandling long-running report generation tasks\nProviding
+ progress indicators and user notifications\nStrategies for canceling or abandoning
+ incomplete reports\nTradeoffs of pre-calculating report data vs. real-time generation\nMaterializing
+ views and denormalizing data for reporting\nExploring event sourcing patterns
+ for reporting needs\nUsing database triggers and stored procedures for reporting\nBalancing
+ data structure optimization for reports vs. day-to-day usage\nCaching report data
+ for faster retrieval and rendering\nCharting and visualization integration in
+ reporting systems\nLinks mentioned:\nPrometheus monitoring system & time series
+ database https://prometheus.io/ \nThinking Elixir \"FLAME with Chris McCord\"
+ https://podcast.thinkingelixir.com/181\nPhoenix LiveView Uploads https://hexdocs.pm/phoenix/fileuploads.html\nhttps://hexdocs.pm/phoenixlive_view/Phoenix.LiveView.UploadWriter.html
+ \nPostgrex PostgreSQL driver for Elixir https://hexdocs.pm/postgrex/Postgrex.html
+ \nEcto https://hexdocs.pm/ecto/Ecto.html \nHeroku cloud application platform
+ https://www.heroku.com/ \nElixir Wizards S9E12 Marcelo Dominguez on Command and
+ Query Responsibility Segregation https://smartlogic.io/podcast/elixir-wizards/s9-e12-marcelo-dominguez-cqrs/\nCommanded
+ Elixir CQRS/ES applications https://github.com/commanded/commanded\nTailwind CSS
+ Framework https://github.com/tailwindlabs\nMemcached https://memcached.org/\nRedis
+ https://redis.io/\nOban https://hexdocs.pm/oban/Oban.html\nETS https://hexdocs.pm/ets/ETS.html\nCapistrano
+ remote server automation and deployment tool https://capistranorb.com/ \n"
+ author: SmartLogic LLC
+ embedUrl: https://fireside.fm/player/v2/IAs5ixts+gRrFRMaV
+ enclosure:
+ url: https://aphid.fireside.fm/d/1437767933/03a50f66-dc5e-4da4-ab6e-31895b6d4c9e/f0d1daa0-bc06-4daf-b6c9-c37a12357821.mp3
+ length: '97997538'
+ type: audio/mpeg
+ itunes:
+ episodeType: full
+ season: '12'
+ author: SmartLogic LLC
+ subtitle: Elixir Wizards Owen and Dan delve into the complexities of building
+ advanced reporting features within software applications. They share personal
+ insights and challenges encountered while developing reporting solutions for
+ user-generated data, leveraging both Elixir/Phoenix and Ruby on Rails.
+ duration: '50:21'
+ explicit: 'no'
+ keywords: software development, web development, programming languages, software
+ engineering, data reporting, app development, user experience, Ruby on Rails,
+ Elixir programming, Phoenix framework, application design, system architecture,
+ API development, real-time applications, open-source software, software design
+ principles, user interface design, project management, tech industry trends,
+ agile methodology, database management, SQL, NoSQL, cloud computing, frontend
+ development, backend development, full-stack development, JavaScript, Python,
+ HTML, CSS, data visualization, machine learning, artificial intelligence, software
+ testing, continuous integration, deployment strategies, version control, Git,
+ GitHub, DevOps, security best practices, network architecture, data science,
+ data analysis, big data, software documentation, coding best practices, code
+ refactoring, system scalability, performance optimization, server-side scripting,
+ client-side scripting, responsive design, mobile app development, cross-platform
+ development, tech startups, tech news, digital transformation, IT management,
+ tech conferences, programming tutorials, coding challenges, software career
+ development
+ image: https://assets.fireside.fm/file/fireside-images/podcasts/images/0/03a50f66-dc5e-4da4-ab6e-31895b6d4c9e/episodes/f/f0d1daa0-bc06-4daf-b6c9-c37a12357821/cover.jpg
+ summary: "\n In today's episode, Elixir Wizards Owen and Dan delve
+ into the complexities of building advanced reporting features within software
+ applications. They share personal insights and challenges encountered while
+ developing reporting solutions for user-generated data, leveraging both Elixir/Phoenix
+ and Ruby on Rails.
\n\nThe discussion zeroes in on crucial data modeling
+ and architectural decisions that enhance reporting efficiency and flexibility.
+ Owen and Dan explore tactics like materialized views, event sourcing, and database
+ triggers to optimize data handling while being mindful of UX elements like progress
+ indicators and background job management.
\n\nThey share insights on leveraging
+ the Elixir/Beam ecosystem’s strengths—like concurrency and streamlined deployment—to
+ tackle common reporting, caching, and integration challenges. The episode highlights
+ the impact of reporting features across all aspects of a software application’s
+ design and architecture.
\n\nKey topics discussed in this episode:
\n\n\n- Reporting
+ on assessment data, survey results, and user metrics
\n- Differences between
+ reporting and performance/error monitoring
\n- Implementing reporting
+ in Elixir/Phoenix vs. Ruby on Rails
\n- Displaying reports in web, printable,
+ PDF, SVG, and CSV formats
\n- Challenges of generating PDFs for large
+ data sets
\n- Streaming CSV data directly to the client
\n- Handling
+ long-running report generation tasks
\n- Providing progress indicators
+ and user notifications
\n- Strategies for canceling or abandoning incomplete
+ reports
\n- Tradeoffs of pre-calculating report data vs. real-time generation
\n- Materializing
+ views and denormalizing data for reporting
\n- Exploring event sourcing
+ patterns for reporting needs
\n- Using database triggers and stored procedures
+ for reporting
\n- Balancing data structure optimization for reports vs.
+ day-to-day usage
\n- Caching report data for faster retrieval and rendering
\n- Charting
+ and visualization integration in reporting systems
\n
\n\nLinks
+ mentioned:
\n\nPrometheus monitoring system & time series database
+ https://prometheus.io/
+
\nThinking Elixir "FLAME with Chris McCord" https://podcast.thinkingelixir.com/181
\nPhoenix LiveView
+ Uploads https://hexdocs.pm/phoenix/file_uploads.html
\nhttps://hexdocs.pm/phoenix_live_view/Phoenix.LiveView.UploadWriter.html
+
\nPostgrex PostgreSQL driver for Elixir https://hexdocs.pm/postgrex/Postgrex.html
\nEcto https://hexdocs.pm/ecto/Ecto.html
+
\nHeroku cloud application platform https://www.heroku.com/
\nElixir Wizards S9E12 Marcelo
+ Dominguez on Command and Query Responsibility Segregation https://smartlogic.io/podcast/elixir-wizards/s9-e12-marcelo-dominguez-cqrs/
\nCommanded
+ Elixir CQRS/ES applications https://github.com/commanded/commanded
\nTailwind CSS
+ Framework https://github.com/tailwindlabs
\nMemcached
+ https://memcached.org/
\nRedis
+ https://redis.io/
\nOban
+ https://hexdocs.pm/oban/Oban.html
\nETS
+ https://hexdocs.pm/ets/ETS.html
\nCapistrano
+ remote server automation and deployment tool https://capistranorb.com/
\n "
+ contentEncoded: "\n In today's episode, Elixir Wizards Owen and Dan
+ delve into the complexities of building advanced reporting features within software
+ applications. They share personal insights and challenges encountered while developing
+ reporting solutions for user-generated data, leveraging both Elixir/Phoenix and
+ Ruby on Rails.
\n\nThe discussion zeroes in on crucial data modeling and
+ architectural decisions that enhance reporting efficiency and flexibility. Owen
+ and Dan explore tactics like materialized views, event sourcing, and database
+ triggers to optimize data handling while being mindful of UX elements like progress
+ indicators and background job management.
\n\nThey share insights on leveraging
+ the Elixir/Beam ecosystem’s strengths—like concurrency and streamlined deployment—to
+ tackle common reporting, caching, and integration challenges. The episode highlights
+ the impact of reporting features across all aspects of a software application’s
+ design and architecture.
\n\nKey topics discussed in this episode:
\n\n\n- Reporting
+ on assessment data, survey results, and user metrics
\n- Differences between
+ reporting and performance/error monitoring
\n- Implementing reporting in
+ Elixir/Phoenix vs. Ruby on Rails
\n- Displaying reports in web, printable,
+ PDF, SVG, and CSV formats
\n- Challenges of generating PDFs for large data
+ sets
\n- Streaming CSV data directly to the client
\n- Handling long-running
+ report generation tasks
\n- Providing progress indicators and user notifications
\n- Strategies
+ for canceling or abandoning incomplete reports
\n- Tradeoffs of pre-calculating
+ report data vs. real-time generation
\n- Materializing views and denormalizing
+ data for reporting
\n- Exploring event sourcing patterns for reporting needs
\n- Using
+ database triggers and stored procedures for reporting
\n- Balancing data
+ structure optimization for reports vs. day-to-day usage
\n- Caching report
+ data for faster retrieval and rendering
\n- Charting and visualization integration
+ in reporting systems
\n
\n\nLinks mentioned:
\n\nPrometheus
+ monitoring system & time series database https://prometheus.io/
\nThinking Elixir "FLAME
+ with Chris McCord" https://podcast.thinkingelixir.com/181
\nPhoenix
+ LiveView Uploads https://hexdocs.pm/phoenix/file_uploads.html
\nhttps://hexdocs.pm/phoenix_live_view/Phoenix.LiveView.UploadWriter.html
+
\nPostgrex PostgreSQL driver for Elixir https://hexdocs.pm/postgrex/Postgrex.html
\nEcto https://hexdocs.pm/ecto/Ecto.html
\nHeroku cloud application
+ platform https://www.heroku.com/
+
\nElixir Wizards S9E12 Marcelo Dominguez on Command and Query Responsibility
+ Segregation https://smartlogic.io/podcast/elixir-wizards/s9-e12-marcelo-dominguez-cqrs/
\nCommanded
+ Elixir CQRS/ES applications https://github.com/commanded/commanded
\nTailwind CSS
+ Framework https://github.com/tailwindlabs
\nMemcached
+ https://memcached.org/
\nRedis
+ https://redis.io/
\nOban
+ https://hexdocs.pm/oban/Oban.html
\nETS
+ https://hexdocs.pm/ets/ETS.html
\nCapistrano
+ remote server automation and deployment tool https://capistranorb.com/
\n "
- title: '"Whose Tailwind is it Anyway?" with Ava Slivkoff'
slug: s12-e04-software-design-development-collab
link: https://smartlogic.io/podcast/elixir-wizards/s12-e04-software-design-development-collab
diff --git a/_data/elixir_wizards_seasons.yml b/_data/elixir_wizards_seasons.yml
index 8265f9da..26a51125 100644
--- a/_data/elixir_wizards_seasons.yml
+++ b/_data/elixir_wizards_seasons.yml
@@ -6155,6 +6155,176 @@ Season 1:
uses Phoenix and Elixir.\n
\ "
Season 12:
+- title: '"Saga of a Gnarly Report" with Owen and Dan'
+ slug: s12-e05-saga-of-a-gnarly-report
+ link: https://smartlogic.io/podcast/elixir-wizards/s12-e05-saga-of-a-gnarly-report
+ guid: f0d1daa0-bc06-4daf-b6c9-c37a12357821
+ pubDate: Thu, 18 Apr 2024 07:00:00 -0400
+ pubDateFriendly: April 18, 2024
+ description: "In today's episode, Elixir Wizards Owen and Dan delve into the complexities
+ of building advanced reporting features within software applications. They share
+ personal insights and challenges encountered while developing reporting solutions
+ for user-generated data, leveraging both Elixir/Phoenix and Ruby on Rails.\nThe
+ discussion zeroes in on crucial data modeling and architectural decisions that
+ enhance reporting efficiency and flexibility. Owen and Dan explore tactics like
+ materialized views, event sourcing, and database triggers to optimize data handling
+ while being mindful of UX elements like progress indicators and background job
+ management.\nThey share insights on leveraging the Elixir/Beam ecosystem’s strengths—like
+ concurrency and streamlined deployment—to tackle common reporting, caching, and
+ integration challenges. The episode highlights the impact of reporting features
+ across all aspects of a software application’s design and architecture.\nKey topics
+ discussed in this episode:\nReporting on assessment data, survey results, and
+ user metrics\nDifferences between reporting and performance/error monitoring\nImplementing
+ reporting in Elixir/Phoenix vs. Ruby on Rails\nDisplaying reports in web, printable,
+ PDF, SVG, and CSV formats\nChallenges of generating PDFs for large data sets\nStreaming
+ CSV data directly to the client\nHandling long-running report generation tasks\nProviding
+ progress indicators and user notifications\nStrategies for canceling or abandoning
+ incomplete reports\nTradeoffs of pre-calculating report data vs. real-time generation\nMaterializing
+ views and denormalizing data for reporting\nExploring event sourcing patterns
+ for reporting needs\nUsing database triggers and stored procedures for reporting\nBalancing
+ data structure optimization for reports vs. day-to-day usage\nCaching report data
+ for faster retrieval and rendering\nCharting and visualization integration in
+ reporting systems\nLinks mentioned:\nPrometheus monitoring system & time series
+ database https://prometheus.io/ \nThinking Elixir \"FLAME with Chris McCord\"
+ https://podcast.thinkingelixir.com/181\nPhoenix LiveView Uploads https://hexdocs.pm/phoenix/fileuploads.html\nhttps://hexdocs.pm/phoenixlive_view/Phoenix.LiveView.UploadWriter.html
+ \nPostgrex PostgreSQL driver for Elixir https://hexdocs.pm/postgrex/Postgrex.html
+ \nEcto https://hexdocs.pm/ecto/Ecto.html \nHeroku cloud application platform
+ https://www.heroku.com/ \nElixir Wizards S9E12 Marcelo Dominguez on Command and
+ Query Responsibility Segregation https://smartlogic.io/podcast/elixir-wizards/s9-e12-marcelo-dominguez-cqrs/\nCommanded
+ Elixir CQRS/ES applications https://github.com/commanded/commanded\nTailwind CSS
+ Framework https://github.com/tailwindlabs\nMemcached https://memcached.org/\nRedis
+ https://redis.io/\nOban https://hexdocs.pm/oban/Oban.html\nETS https://hexdocs.pm/ets/ETS.html\nCapistrano
+ remote server automation and deployment tool https://capistranorb.com/ \n"
+ author: SmartLogic LLC
+ embedUrl: https://fireside.fm/player/v2/IAs5ixts+gRrFRMaV
+ enclosure:
+ url: https://aphid.fireside.fm/d/1437767933/03a50f66-dc5e-4da4-ab6e-31895b6d4c9e/f0d1daa0-bc06-4daf-b6c9-c37a12357821.mp3
+ length: '97997538'
+ type: audio/mpeg
+ itunes:
+ episodeType: full
+ season: '12'
+ author: SmartLogic LLC
+ subtitle: Elixir Wizards Owen and Dan delve into the complexities of building
+ advanced reporting features within software applications. They share personal
+ insights and challenges encountered while developing reporting solutions for
+ user-generated data, leveraging both Elixir/Phoenix and Ruby on Rails.
+ duration: '50:21'
+ explicit: 'no'
+ keywords: software development, web development, programming languages, software
+ engineering, data reporting, app development, user experience, Ruby on Rails,
+ Elixir programming, Phoenix framework, application design, system architecture,
+ API development, real-time applications, open-source software, software design
+ principles, user interface design, project management, tech industry trends,
+ agile methodology, database management, SQL, NoSQL, cloud computing, frontend
+ development, backend development, full-stack development, JavaScript, Python,
+ HTML, CSS, data visualization, machine learning, artificial intelligence, software
+ testing, continuous integration, deployment strategies, version control, Git,
+ GitHub, DevOps, security best practices, network architecture, data science,
+ data analysis, big data, software documentation, coding best practices, code
+ refactoring, system scalability, performance optimization, server-side scripting,
+ client-side scripting, responsive design, mobile app development, cross-platform
+ development, tech startups, tech news, digital transformation, IT management,
+ tech conferences, programming tutorials, coding challenges, software career
+ development
+ image: https://assets.fireside.fm/file/fireside-images/podcasts/images/0/03a50f66-dc5e-4da4-ab6e-31895b6d4c9e/episodes/f/f0d1daa0-bc06-4daf-b6c9-c37a12357821/cover.jpg
+ summary: "\n In today's episode, Elixir Wizards Owen and Dan delve
+ into the complexities of building advanced reporting features within software
+ applications. They share personal insights and challenges encountered while
+ developing reporting solutions for user-generated data, leveraging both Elixir/Phoenix
+ and Ruby on Rails.
\n\nThe discussion zeroes in on crucial data modeling
+ and architectural decisions that enhance reporting efficiency and flexibility.
+ Owen and Dan explore tactics like materialized views, event sourcing, and database
+ triggers to optimize data handling while being mindful of UX elements like progress
+ indicators and background job management.
\n\nThey share insights on leveraging
+ the Elixir/Beam ecosystem’s strengths—like concurrency and streamlined deployment—to
+ tackle common reporting, caching, and integration challenges. The episode highlights
+ the impact of reporting features across all aspects of a software application’s
+ design and architecture.
\n\nKey topics discussed in this episode:
\n\n\n- Reporting
+ on assessment data, survey results, and user metrics
\n- Differences between
+ reporting and performance/error monitoring
\n- Implementing reporting
+ in Elixir/Phoenix vs. Ruby on Rails
\n- Displaying reports in web, printable,
+ PDF, SVG, and CSV formats
\n- Challenges of generating PDFs for large
+ data sets
\n- Streaming CSV data directly to the client
\n- Handling
+ long-running report generation tasks
\n- Providing progress indicators
+ and user notifications
\n- Strategies for canceling or abandoning incomplete
+ reports
\n- Tradeoffs of pre-calculating report data vs. real-time generation
\n- Materializing
+ views and denormalizing data for reporting
\n- Exploring event sourcing
+ patterns for reporting needs
\n- Using database triggers and stored procedures
+ for reporting
\n- Balancing data structure optimization for reports vs.
+ day-to-day usage
\n- Caching report data for faster retrieval and rendering
\n- Charting
+ and visualization integration in reporting systems
\n
\n\nLinks
+ mentioned:
\n\nPrometheus monitoring system & time series database
+ https://prometheus.io/
+
\nThinking Elixir "FLAME with Chris McCord" https://podcast.thinkingelixir.com/181
\nPhoenix LiveView
+ Uploads https://hexdocs.pm/phoenix/file_uploads.html
\nhttps://hexdocs.pm/phoenix_live_view/Phoenix.LiveView.UploadWriter.html
+
\nPostgrex PostgreSQL driver for Elixir https://hexdocs.pm/postgrex/Postgrex.html
\nEcto https://hexdocs.pm/ecto/Ecto.html
+
\nHeroku cloud application platform https://www.heroku.com/
\nElixir Wizards S9E12 Marcelo
+ Dominguez on Command and Query Responsibility Segregation https://smartlogic.io/podcast/elixir-wizards/s9-e12-marcelo-dominguez-cqrs/
\nCommanded
+ Elixir CQRS/ES applications https://github.com/commanded/commanded
\nTailwind CSS
+ Framework https://github.com/tailwindlabs
\nMemcached
+ https://memcached.org/
\nRedis
+ https://redis.io/
\nOban
+ https://hexdocs.pm/oban/Oban.html
\nETS
+ https://hexdocs.pm/ets/ETS.html
\nCapistrano
+ remote server automation and deployment tool https://capistranorb.com/
\n "
+ contentEncoded: "\n In today's episode, Elixir Wizards Owen and Dan
+ delve into the complexities of building advanced reporting features within software
+ applications. They share personal insights and challenges encountered while developing
+ reporting solutions for user-generated data, leveraging both Elixir/Phoenix and
+ Ruby on Rails.
\n\nThe discussion zeroes in on crucial data modeling and
+ architectural decisions that enhance reporting efficiency and flexibility. Owen
+ and Dan explore tactics like materialized views, event sourcing, and database
+ triggers to optimize data handling while being mindful of UX elements like progress
+ indicators and background job management.
\n\nThey share insights on leveraging
+ the Elixir/Beam ecosystem’s strengths—like concurrency and streamlined deployment—to
+ tackle common reporting, caching, and integration challenges. The episode highlights
+ the impact of reporting features across all aspects of a software application’s
+ design and architecture.
\n\nKey topics discussed in this episode:
\n\n\n- Reporting
+ on assessment data, survey results, and user metrics
\n- Differences between
+ reporting and performance/error monitoring
\n- Implementing reporting in
+ Elixir/Phoenix vs. Ruby on Rails
\n- Displaying reports in web, printable,
+ PDF, SVG, and CSV formats
\n- Challenges of generating PDFs for large data
+ sets
\n- Streaming CSV data directly to the client
\n- Handling long-running
+ report generation tasks
\n- Providing progress indicators and user notifications
\n- Strategies
+ for canceling or abandoning incomplete reports
\n- Tradeoffs of pre-calculating
+ report data vs. real-time generation
\n- Materializing views and denormalizing
+ data for reporting
\n- Exploring event sourcing patterns for reporting needs
\n- Using
+ database triggers and stored procedures for reporting
\n- Balancing data
+ structure optimization for reports vs. day-to-day usage
\n- Caching report
+ data for faster retrieval and rendering
\n- Charting and visualization integration
+ in reporting systems
\n
\n\nLinks mentioned:
\n\nPrometheus
+ monitoring system & time series database https://prometheus.io/
\nThinking Elixir "FLAME
+ with Chris McCord" https://podcast.thinkingelixir.com/181
\nPhoenix
+ LiveView Uploads https://hexdocs.pm/phoenix/file_uploads.html
\nhttps://hexdocs.pm/phoenix_live_view/Phoenix.LiveView.UploadWriter.html
+
\nPostgrex PostgreSQL driver for Elixir https://hexdocs.pm/postgrex/Postgrex.html
\nEcto https://hexdocs.pm/ecto/Ecto.html
\nHeroku cloud application
+ platform https://www.heroku.com/
+
\nElixir Wizards S9E12 Marcelo Dominguez on Command and Query Responsibility
+ Segregation https://smartlogic.io/podcast/elixir-wizards/s9-e12-marcelo-dominguez-cqrs/
\nCommanded
+ Elixir CQRS/ES applications https://github.com/commanded/commanded
\nTailwind CSS
+ Framework https://github.com/tailwindlabs
\nMemcached
+ https://memcached.org/
\nRedis
+ https://redis.io/
\nOban
+ https://hexdocs.pm/oban/Oban.html
\nETS
+ https://hexdocs.pm/ets/ETS.html
\nCapistrano
+ remote server automation and deployment tool https://capistranorb.com/
\n "
- title: '"Whose Tailwind is it Anyway?" with Ava Slivkoff'
slug: s12-e04-software-design-development-collab
link: https://smartlogic.io/podcast/elixir-wizards/s12-e04-software-design-development-collab
diff --git a/_data/elixir_wizards_transcripts.yml b/_data/elixir_wizards_transcripts.yml
index 0df1f6f1..2cdb698d 100644
--- a/_data/elixir_wizards_transcripts.yml
+++ b/_data/elixir_wizards_transcripts.yml
@@ -269,3 +269,5 @@ s12-e03-background-jobs-web-development:
English: "/podcast/elixir-wizards/transcripts/s12-e03-background-jobs-web-development.txt"
s12-e04-software-design-development-collab:
English: "/podcast/elixir-wizards/transcripts/s12-e04-software-design-development-collab.txt"
+s12-e05-saga-of-a-gnarly-report:
+ English: "/podcast/elixir-wizards/transcripts/s12-e05-saga-of-a-gnarly-report.txt"
diff --git a/podcast/elixir-wizards/transcripts/s12-e05-saga-of-a-gnarly-report.txt b/podcast/elixir-wizards/transcripts/s12-e05-saga-of-a-gnarly-report.txt
new file mode 100644
index 00000000..8fac4588
--- /dev/null
+++ b/podcast/elixir-wizards/transcripts/s12-e05-saga-of-a-gnarly-report.txt
@@ -0,0 +1 @@
+S12E05 Saga of a Gnarly Report
[00:00:00]
[00:00:11] Owen: Hello everyone. I'm Owen Bickford, Senior Software Developer at SmartLogic.
[00:00:16] Dan: And I'm Dan Ivovich, Director of Engineering at SmartLogic, and we'll be your hosts for today's episode. For episode five, we're talking about building reports in some interesting projects. So let's talk about reports, Owen.
[00:00:30] Owen: I get A's every time. I've never failed. So is that kind of report,
[00:00:34] Dan: Yeah, sure. Yeah. What are we, what are we reporting on? I guess is, uh, maybe a question.
[00:00:39] Owen: Right. So, we're a we're, software agency, and we get a wide, actually pretty wide variety of client projects. You know that better than I do. The one that I was thinking about when we were drafting this episode was a client project on that I'm on currently for assessment data. And I think there's a couple of different applications that we have that have different kinds of forms and formats of assessment data. Is that fair to say?
[00:01:08] Dan: Yeah, I mean, I think when I think about reporting. Usually it is like survey or multiple choice or some sort of scale that things are being reported or captured on and then need to be reported with . Um, and so in a sense it is aggregate of what's the average or, you know, and then segmenting that by other characteristics you have who or what the data is But then also, you know, sometimes we don't do a lot of like what I consider time series reporting. It doesn't usually come up for most of our, client projects. But time is sometimes relevant even with just assessment data , because you have assessments that take place over time.
[00:01:47] And, so, in the very least you're filtering often on tell me about assessments that took place within this window. So, yeah, those are
[00:01:55] the kinds of reports.
[00:01:56] Owen: and then just to call out a couple of things we're not talking about, we're not talking about performance monitoring, , error reporting, logging, that kind of thing. This is, these are application features for analyzing the performance of users. Even that's like confusing things. I think, uh, it's, it's about,
[00:02:17] Dan: It's about the end user entered data , not about the performance of the application itself.
[00:02:21] Owen: Yeah. Results of assessments and aggregate data
[00:02:24] Dan: about that. Surveys, assessments, whatever you to call them. It's often what we're thinking about.
[00:02:28]
[00:02:28] And I think we're coming at this from slightly different perspectives. I'm working, I've been working on a, I would say a moderately sized Elixir Phoenix application, uh, using a lot of LiveView to collect the data and also to build the reports and display the data. Can you tell me a little bit about the projects you've worked on that are taking things a
[00:02:47] Owen: little bit differently?
[00:02:48] Dan: The, two that come to mind the most, are Similar in the type of data they're gathering and like kind of how it gets captured, , but are both in Ruby and Ruby on Rails. You know, and then I guess there are some other things that we do for, other types of data, but they don't have large reporting components to them . and then there is one other one that's another Rails app that we do, that, is Self assessment data. So still kind of assessment data, but like, you know, these are my goals and this is what I want to do. And, so that stuff kind of plays into that aspect.
[00:03:23] Owen: Right. And I can think of apps that I use or apps that I've seen marketed on places. So like any kind of fitness application is going to have some kind of reporting about whatever data it's collecting. Medical apps will often have some kind of reporting as well. And they're gated usually over different time windows.
[00:03:42] So sometimes two days, seven days, a month, 90 days, that kind of thing. Are there others that come to your mind? Like apps that we haven't worked on that might also involve some kind of reporting?
[00:03:52] Dan: Well, I mean, we do use Prometheus for performance capturing of various things. And so that's just like super heavy time series metric type data, labeled and annotated. , and so while we don't necessarily work on that as a core product, it is a product that exists that we leverage pretty heavily.
[00:04:08] you could argue, like, We do have some like push notification systems. That's a type of reporting. You know, when a certain data exists and meets criteria , send a notification, kind of your, your mention of like health and fitness made me think of that. So we have a few of those ,, where it's that, even if we're considering notifications, then we have a bunch of apps that, , given an event that happens, is there, uh, an email that gets sent or some sort of summary at the end of the week about, like, what happened as a type of report?
[00:04:35] Not what you and I were probably thinking about when we came up with this topic, but, still a type of reporting that you need to deal with.
[00:04:41] Owen: Just to scope this down a little bit, I think we'll focus on that, the type of aggregate data that we're kind of number crunching and then putting onto a, uh, an asset somewhere or like a artifact. So there are different formats that we've supported in the application I'm working on, the client project.
[00:04:58] So, First and foremost, we're building everything to be displayed on the web with maybe a couple of exceptions. Even just on the web, we also like those to be printable. And there've been a couple of different approaches to making views within our application printable. And that's all HTML. And then, we also have been generating the same results and CSV format, so you can download a file, to get just the pure raw data, the aggregate data, at least.
[00:05:26] Dan: Yeah, that's always my favorite, right? Just give the end user a CSV file of a lot of stuff and let them figure out what they want to do from there.
[00:05:33] Owen: They're going to go to Excel anyway. So, uh, and then I know we've. We've been aspiring to generate PDFs because of the madness of printing, so that we can have more consistency. That is its own saga, PDF generation. Are there some other formats that you've had to interact with or produce?
[00:05:54] Dan: I mean, I think in one of our larger ones, those are the main formats, CSV, HTML, and PDF, but I think what's interesting about the approach that we took is, there's a lot of kind of pre calculation to get the report right. And it's a lot of data gathering calculation. And it's more than is currently handled by the database, which, you know, has its own disadvantages from a data optimization standpoint.
[00:06:14] But then the end result is we serialize all of the data for the powers of report into an intermediate format. And then the neat thing of that is at that point, then the client can decide. What type of data they want and the report's already generated. So you can get an HTML preview and then say, yeah, that looks good.
[00:06:31] So give me the CSV or give me the PDF. And the system is only just then re rendering based on that pre calculated data and we don't have to go through the whole generation process again.
[00:06:41] Owen: Right. And one saga, well, one part of the saga that we went through, I think initially when we were producing CSVs, the first pass was to generate, CSV files on the server. So we'd create a temporary directory, do all this number calculation on the server, write all the rows into a CSV file, and I think this is all happening synchronously.
[00:07:01] So then once the file is ready, it, is presented to the browser and I think it ... I think it was just a download link actually, now that I think about it. , one thing I was excited about, I remember hearing in another podcast, I don't remember which one. Um, I think it was Chris McCord talking about the idea, and I think they were maybe even spitballing about the idea of streaming some of the LiveView developments that you could actually stream contents of a file directly into the browser.
[00:07:30] So you don't need to temporary file in the first place. It took me several months to A, get to the point where we had space to work on it, but also B, to kind of think through How to actually build that. And so, , I'm really happy that like we have a, a live component in our system, it's called File Writer.
[00:07:47] And basically what it does is, you know, it, it interacts with the Oban job, and some pub sub stuff in our system. But as soon as the data for the report is ready, so it's gone through the SQL, , Ecto, uh, to gather, gather all the data, and it's gone through maybe some reducers and maps and everything.
[00:08:04] We've got our kind of end result, our stream of records. So it's not being kept in memory on the server. And the file writer is able to receive and request records out of that stream. And it streams, it like converts it all to, to I think JSON or, just raw binary, and then writes it into the client side through a JavaScript hook, using, you know, LiveView hooks.
[00:08:28] So we get our, our results calculated into a stream. We pluck those rows out of the stream through the file writer, goes into JavaScript, writes into a blob. And then once it has received an empty row, like where there's no more binary to be written, then it writes the file with whatever file name we provided.
[00:08:48] and the user just sees the file appear in their downloads, you know, I'm on Mac, so I get this little animation where this little file floats up to the downloads in Chrome, and then I've got it in my Downloads folder. And all of that happens a lot more transparently. It's all behind the scenes now, whereas it used to be a kind of a multi step thing that the user would have to kind of go through to get the file.
[00:09:09] So, the end result now is you click, you know, Download CSV. And of course, we're not dealing with huge amounts of data yet. So this all happens basically instantly. You hit download CSV and then all of a sudden you instantly see a CSV file shows up in your computer.
[00:09:27] Dan: Yeah, it made me think, um, we've definitely had cases where we would do streaming of CSV files. And CSV works great for streaming because you can send a line at a time. You don't necessarily worry about like where the end is. I think the first time I really thought about it a lot was in those kind of like early Heroku days when you had like a really hard 30 second timeout, unless you opened, like a streaming connection and were still streaming data, then you could go longer.
[00:09:50] And so this was a common case from a reporting perspective of like, okay, well we have a big CSV. It's going to take a while to calculate, and the user's willing to wait for it. So like open, open that stream and then send it, a few lines at a time or a line at a time. The disadvantages you get is like, if you can't, if, as you open the stream, if you don't know how big the file is, then you can't tell the browser how many bytes it's expecting, so it can't tell you anything on terms of progress, like the user has very little indicator as to will this ever end? Like how long will I be waiting? Which isn't the best experience, but it does from a technical standpoint, keeps your memory usage low you're not necessarily piling everything up into memory to then send all at once depending on how you're even generating those rows , and you can Make it feel like it's kind of continually happening instead of there being like a long pause.
[00:10:34] It's just a trade off, I think, to some degree.
[00:10:36] Owen: Yeah, that is something I think we would need to upgrade eventually, with enough data and users and everything in the system is to add more transparency about, if it's going to take several seconds or a minute or more, let the user know what's happening, right, as much as possible.
[00:10:54] And the tricky part there is like, part of that is just the database gathering records based on all this complex query and maybe we can add indexes to speed that up if we need to. So we wouldn't really know even the size of the volume of data until those queries have been executed.
[00:11:13] And then we would at least know if we're talking about like just rows, we would have a row count potentially. And then we could count how many we've received out of the total rows use that as a progress indicator.
[00:11:25] Dan: Well, yeah, and I think the other interesting side of this too, and this came up maybe a little bit when I was talking to Joel about background jobs, but also I've seen it come up in just like long running queries and things like that, is like if you take a long time and the user doesn't wait, there's always a question of like, do you know that they went away and are no longer waiting?
[00:11:41] And if they're not, are you still, you know, burning a bunch of CPU cycles. Are you still tying down your database? Are you still using one of your active workers to figure out what they were looking for before they left? and when you have these long running reports, something that we've had to bake into some of our more complicated report systems that we've had to build is, some amount of like the report as it's gathering data to every some period of rows processed or, you know, data ingested or whatever it is that it's doing to kind of check something that indicates whether the report has been cancelled in some way or should no longer be completed, so that you can give up and free that worker cycle, back to your pool. And that can be true depending on your system and your architecture, even for something that you're live streaming to a web page, just depending on your ability to notice whether a connection's been severed.
[00:12:28] Owen: Yeah, we actually have some really nice tools with LiveView. There's a terminate callback. So. I don't think we've used this yet in our app, but, whenever the LiveView terminates, like the user has been either signed out or they've grown impatient, they've moved to elsewhere, we could use the terminate callback to cancel the job that was created, you know, previously or cancel any jobs that were previously created by the user.
[00:12:53] so we have either very kind of like sledgehammer ways of solving that or kind of scalpel ways of solving it.
[00:12:59] Dan: The, the kind of constant streaming of rows has the advantage, right? Of like, if you were sending those over an open socket and that socket closes, your application may error because it's trying to write to a socket that's closed, but like, that's fine. That's like a happy, let it fail case of good, I'm not wasting any energy anymore.
[00:13:14] But at least you like have the indicator that it stopped. Whereas on the pre calculations, if you're pre calculating all the data, you may not yet know whether they've gone away, depending on what your process model looks like.
[00:13:24] Owen: And one way around that. So like that may also be just depending on the exact, you know, application, the user expectations that may be bad at user experience that I click generate report. I don't want to wait for it. So I go about doing the rest of my job. And then I learned later on that the report never finished because I wasn't waiting for it.
[00:13:45] So I think one, one opportunity for improving UX in these areas is. Having kind of like a global notification system so that we can just operate, background jobs in the, in the background, and then notify the user whenever a result has actually, you know, whenever a job is finished, whatever the result may be.
[00:14:05] Dan: I see that as a very common pattern in tools that we use for the business. I can think of a couple of them offhand where it's like you request an export or a report of some sort and you get, and it says, okay, we'll send you an email when it's done. And actually just today I did that where I picked something, I hit export, it said give me a CSV and, you know, a little notification.
[00:14:22] We'll send you an email when it's done. And then, like, two seconds later, my browser's downloading the file. And I'm like, oh, well, that's confusing, because you kind of did both. I mean, like, I'm happy I don't have to go to my email and get the file, but that's not what you told me you were going to do.
[00:14:34] Um, so, you know, pick, pick a lane and, you know, just be consistent.
[00:14:39] Owen: Yeah, I think even like adding some logic that, okay, if this job is completed after 10 seconds or a minute or whatever, from whenever it was started, then maybe that's an email notification. Otherwise notify PubSub or whatever in the system.
[00:14:55] Dan: Yeah, well, it's something that I think we don't think about a lot in the early design of our reporting systems, but can be very helpful down the line is like, what are my early indicators based on this report? Either the account I'm reporting on, the date range, what are the heuristics I could use to determine is this going to take a while or is this going to be fast?
[00:15:13] I've been in tools that we use where it's like, you know, okay, if I'm looking at a week's worth of data and I say, give me a report, It doesn't tell me it's going to email me. It just gives me the file, right? But if I'm looking at like a year's worth of data and I say, give me a report, then it gives me the banner of like, Hey, we'll send you an email with the link when this thing is ready.
[00:15:29] And so I think there are often those kinds of like indicators to drive a user experience that aligns with the expectation of time, Um, I think, you know, we kind of skimmed over a little bit at the top, like the, the querying and gathering of data, you know, I want to do as much in the database as I can, because like databases are much faster at like pulling things together and, and, you know, summarizing data than, you know, most languages.
[00:15:59] Um, But that's not always possible, depending on how things are structured. And, you know, for applications, some of those that we inherit from other, other developers, or that just were never, like, when they originally, like, thought of, didn't expect to have to query the data in this way. And so there's other solutions that sometimes we have to do there, and I've done a handful of them.
[00:16:19] Uh, sometimes it is a matter of actually restructuring the data, and then, Now you have a maintenance problem of how do I keep this alternate form of the data current? Uh, but if it is, you know, restructured and current, then I can report on that structure faster than the structure that powers the actual day to day usage of the application.
[00:16:40] Um, you know, and that's, I think, where you start to get into systems of like, you know, warehousing data versus live data and how all that works, which is like, now you've added a lot more complexity to your system. But we've also done, you know, the database can almost do some of that for you. Because you can often build a view and just materialize it.
[00:16:58] And so you're telling the database, like, restructure all of your data that looks like this and effectively make a copy with some sort of refresh interval. But it only works if your reporting is allowed to be some amount of time stale and that that refresh is also fast enough and doesn't pull down your database in a way that is detrimental.
[00:17:16] Um, and so we've, you know, depending on the use case, we've had to do, you know, kind of all of those options. Um, but. Yeah. The good news is there's lots of tools in the shed.
[00:17:27] Owen: And those tools get upgrades, right? So the fun, fun thing is you learn that like, Oh, this was really slow and painful whenever we did this in MySQL. And we never want to do that again. Uh, and then you, it's sometimes you'll find yourself working in Postgres and maybe those rules or those pain points aren't exactly the same.
[00:17:50] Uh, or even, you know, you, you do things a certain way in Postgres 9 and then, uh, A few years later, you're in Postgres 15 or 16 or something, and it just has different performance characteristics. So maybe it can do things, uh, better the original, like maybe your original solution would have actually worked on the later version of Postgres.
[00:18:11] Uh, so yeah, the, but the whole point about data structures is something I keep coming back to. I don't think I would want to rewrite the entire app that we've built here, but I Want to, for the project I'm talking about, I would, I don't think it's, it's not a tragic mistake. We've been able to do everything we needed to do so far, but I think it's unfortunate that we went with kind of like a prototype of the data structure for this, uh, some of this data.
[00:18:44] So
[00:18:45] Dan: Mm-Hmm.
[00:18:45] Owen: to talk in the abstract terms, what we're doing currently is, uh, scoring or storing results into, uh, just a, a plain Elixir map. Inside of a field in Ecto, right? So you've got the schema. Exactly. So you've got the schema that represents the kind of high order thing. And then you've got a field within that, that contains the kind of like the entries, and we've got a few of those because there's different types of, it's hard not to use the terms of the actual project.
[00:19:22] I'm not sure if we can, but there's like.
[00:19:24] Dan: we shouldn't,
[00:19:24] Owen: Yeah, there's different kind of like, um, let's call them facets
[00:19:29] that need to be stored within this kind of parent structure. And so we've got three or four different maps that contain that. And, uh, it's worked well enough. I think one thing in hindsight, or if, if there's a similar project that we are working on, the next iteration would be to break those out as, into associations.
[00:19:50] So. Each, each
[00:19:54] thing, right, each, each item within this thing could have its own record. And I think on modern Postgres this would be, you know, it would have indexes on the right column so that it could all be aggregated well. That's one step
[00:20:12] is kind of splitting up the data into separate tables.
[00:20:14] Dan: and we have other projects that. are closer to the architecture I think you're thinking you'd like to move to and like, you know, they started that way and then there's like, there's other disadvantages to that, you know, and so we've actually gone through iterations of restructuring large tables of structured data to, in some cases, denormalize them to a degree.
[00:20:37] So that we can look up things faster on them and create those optimal pieces. And I think the, one of the challenges I always see with reporting is that the structure that makes the most sense to drive your webpage, you know, in the form and the collection is sometimes very different. from how you want to report on that data.
[00:20:55] Um, but I also believe very strongly in like, if you try to optimize for that reporting upfront, like you're just not going to know where you're going to bottleneck or you're not going to know exactly like until you're in it. And so I don't often want us to put a whole lot of effort into figuring all of that out because once we've got the data, like we can always change how the data looks and we can always change how it is structured and we can replicate it and we can, you know, we can do all sorts of things.
[00:21:20] Um, You know, but often the collection is the key part of the business and the reporting is, you know, important but can take longer and it's like sometimes acceptable. Um, so that's kind of where I tend to drive our, like, optimization or at least planning in how we approach it.
[00:21:39] Owen: Yeah, I think transforming the data, I could, I can kind of see how we would get there with, you know, data migrations to kind of pull these things out of these maps and put them into separate tables. The hardest part I think would be refactoring all the code that expects it to be in maps with the original record.
[00:21:57] So, uh, that would, that would be the big lift, right? If, you know, if this was a real problem that we needed to solve. I think this
[00:22:07] also kind of.
[00:22:08] Dan: we could probably introduce an abstraction. That would, that would be the, you know, introduce an abstraction that just maps through right now and then use that to then update how it maps across. And, you know, it could be done and it might take a couple of deploys, but it could be done.
[00:22:24] Um,
[00:22:28] Owen: point that you ran into with that approach was mostly on the reporting side, is that right? Or was it more on the inserts side?
[00:22:38] Dan: yeah, it was, it was kind of a combination. So I think that the structure that we moved into is about the same in terms of inserts, but we were actually even just having like, Viewing the data in the web page, like not even reporting, but just like, here's a table, right? Like just normal operation. Here's a table and you need to know something about it.
[00:22:55] Um, And so it was a little bit of like, well, I don't want to have to go look at all of my associations just to get this one summary data point. Right. And I think like so much about like being dry and not repeating yourself and like optimal data storage and like normalizing a database so that you don't have a data that's repeated, like would tell you, don't keep a summary of all your associations in your parent object.
[00:23:23] Right. But sometimes you want that. Um, and as long as you have like some level, some reason to maintain the confidence of that data will be accurate, that I can't make a change to some association without updating that parent that is the summary, then it can have a real advantage to have that summary column, uh, about all of the nested data such that you can report on it, filter on it, just quickly load and load and load effectively a shallow object.
[00:23:53] Uh, that has that data and then only go get the rest of it when you truly need it.
[00:23:58] Owen: Yeah, I think that's Yeah. So with, we, we are in a semi happy medium, right? So like the, I think they're very original. If I'm rewinding back a year to when this project started, I think the original idea was to have a nested map of, you know, a hierarchy, there's like a, it's a hierarchical data structure ultimately that we're representing.
[00:24:22] And so you would have these kinds of multiple tiers of, of these entities, and then each one would have its, you know, different fields that it needs. Uh, that, like, just representing that in a, just rendering a form that could actually interact with that was going to be a, there's just a lot of, you know, sand in the gears to be able to
[00:24:44] Dan: Yeah.
[00:24:45] And that's the tradeoff, right? Is like, what format is good for one isn't necessarily good for the other, and you're having to kind of, to draw that, draw that line somewhere.
[00:24:56] Owen: Yeah. So like when you say flat objects, that's, I think that's, that's what's made what we have now. workable for, for the year that we've been working on it. Like we have, instead of like one big map with, you know, multiple levels within it, we've got three or four maps that contain just the key values and the values are just simple values, you know, strings, integers, maybe a list of strings, and that's as complicated as it gets in there.
[00:25:27] Um, But thinking about, so I think materialized views are one way to kind of store the data in the more kind of normalized, separate, associated table kind of format. And then kind of generate materialized views that represent, uh, recently accurate or, you know, eventually consistent, uh, you know, aggregate information about that data.
[00:25:57] I think the next step from there would be something like event sourcing. So, and I, I think I've, I dabbled with this, uh, about a year ago after getting a little bit of confidence and, you know, just the vocabulary of event sourcing, which was, it's, it's intimidating at first, and this isn't an episode about event sourcing, but, uh, I think it, it's a paradigm that, that could pair very nicely with LiveVue because in LiveVue, uh, You know, you might have a form on the page with hundreds of items.
[00:26:29] And then instead of making that one big form, you can, might have, uh, listeners or events that are triggered from individual elements on the page. And dispatching those as very atomic records and rights, uh, could be, uh, I think, uh, a nice way to have a scalable system and then, uh, kind of like PubSub, it allows you to.
[00:26:58] PubSub is actually part of the equation, I think, but once you dispatch this event that, like, this particular type of data has been changed or inserted or created or whatever, then it can have all these ripple effects of now it's going to update this aggregate, this other aggregate. And I like the idea. I don't, haven't implemented this in practice for this type of application yet, but I like the idea of like an event changed and it will eventually result in a report that is ready to go.
[00:27:29] It's like pre calculated before anyone even demands it.
[00:27:32] Dan: Yeah, and we have some things that are, well, not like, I would say like, you know. event sourcing, capital E, capital S, register trademark. You know, they're like the idea of, hey, given this data or this change in data, let me go update the rows. Let me go do those pre calculations that make sense. Right. And when, and then, you know, the, when we roll something like that out, then the kind of first step is to run a job that goes and takes all the existing data and effectively plays it into your event.
[00:28:01] stream and says like, okay, like pretend all these things are happening so that you're, the end result is, is what it should be. And, you know, I, I think where, where that stuff gets intimidating to some people is like, you know, okay, well now, how can I have race conditions? Like what, what are the order in which these things happen?
[00:28:20] And the order often then matters, or, you know, what if, what if I miss something? Um, and I think we're all a little, maybe a little traumatized by the, uh, callbacks in, in certain ORMs, uh, that would be like, okay, well, when I update data here, go update data all these other places. And then you end up with this web that you can't untangle.
[00:28:38] And, you know, but if you ever break the web or if you miss a piece, then you can have inconsistency in your data in a way that isn't, isn't great. And it may be that there's now been some progression in event sourcing, like frameworks and tooling that some of this can be kind of worked around. Um, but I think for, uh, For a new application, when you're still kind of like unwinding the problem space, uh, I don't know, like, maybe you can figure out the event sourcing from the start.
[00:29:05] Maybe you can't, um, you probably can. Um, but I think it's just, it's not a tool that I see a lot of people reaching to. You know, it's not something I think gets talked about enough and it's something I probably, you know, we probably should dig deeper into at some point. Um, cause I think it has some cool, uh, downstream impact on some of the things that we find ourselves having to do pretty often.
[00:29:26] Owen: Yeah. Shout out to Commanded. I, uh, I think I was actually on a couple of episodes where we talked to different folks who had used Commanded, uh, or done some form of event sourcing as well.
[00:29:40] Dan: Yeah, we'll have to link some back catalogue episodes there,
[00:29:43] Owen: right. we talked to somebody, yeah. Yeah,
[00:29:54] and, but they also mentioned that you don't have to buy into the full ES ecosystem. You can kind of take some of these paradigms and like, just start breaking your data apart a little bit. And they, I think the, the thing that stuck with me from that conversation is that, uh, It kind of goes back to the status column problem, right?
[00:30:14] So, and a lot of your records, you need a status column. So, you know, whether this thing is, uh, if we're talking about jobs, we're talking about like pending, in progress, completed, failed, canceled, that kind of thing, uh, and then, and you'll see this in Obon, you'll see timestamp columns for every status, right?
[00:30:32] So you'll see, uh, failed at has a timestamp value, uh, completed at has a timestamp value. If you were to represent that with event sourcing, you would have your, you know, your status would be your current status, but then you'd have a separate table, uh, of all the events. So you would see the progression of events that happened with the, with timestamps for whichever record you're talking about.
[00:30:58] So you'd see it started in progress, then it went to, uh, completed or then it paused and then it went back to completed or in progress or whatever. Uh, so that all sounds very enticing to me as far as having more transparency about kind of how and when things happened. Um, I think the one trade off that I've heard from, from people who've worked in these systems is that debugging.
[00:31:24] Uh, so how did, if I'm looking at an aggregate, how did the data get to this point, uh, or if there's an error, I think that's actually more of where the, some of the pain points are. So if you're looking at an aggregate, you can go look at the events and you can rebuild the thing from history. But I think, uh, debugging errors maybe is a little bit more convoluted because you've got an event that happened, but you don't necessarily know, uh, without, you've got this web.
[00:31:50] I think you're talking about like, we've got to kind of like figure out where this event came from, uh, if something goes wrong, but I do, yeah, so I think there's, I think there's still some value in either using the tools that commanded or, uh, splitting some of this data up so that it can be, uh, so that the querying on the reporting side is simplified. Either we're kind of like pushing data to a aggregate or a materialized view, uh, or our reports are able to kind of calculate, like they're able to pull smaller sets of records. One thing we have, one potential problem we have right now is, uh, uh, if we wanted to look at a subset of, uh, values within this parent schema, we have to pull in the whole schema and we can't like report on just the sub items within there.
[00:32:43] So, uh, it would be nice, you know, that. May not be a problem, but it could be a problem once there's, you know, sufficient number of users or data in the system.
[00:32:53] Dan: I think on similar products that I've worked on, well, okay, so in one, the granularity of the data and how it fits, it was numerical, and so like, a standard deviation across. a set of data was important, um, or an average across a set of data was important at the kind of like most granular level. And that's actually the project where we do the most materialization of views so that we have like, you know, very firm, indexable, like usable data, um, to be able to do those queries quickly in the product that's, I think, most similar to what you're thinking of and what you've been working on recently.
[00:33:31] We almost never need to go down to that granular level. When we're doing reporting, it is almost always at the like, summary level of like, what was the full result, not how did that result come to be. And so that does allow us then to use that kind of like, bookkeeping on the parent object of like, the underlying data is there, but I don't need to replay it, which is like, almost like an event sourcing paradigm to a degree of like, here are all the results, but all I care about is the aggregate result.
[00:33:59] And that's true at that level, but then you have the potential other level of, well. Over how many entries, over what time, for which accounts, for which other users are related to those things. And that's where it can get really, really complicated. And so the more filter criteria, the more grouping criteria that you empower your users with, in a way, the more burdensome your reporting can be, and then it gets to be difficult to optimize every single path through all of that.
[00:34:29] Owen: I'm thinking, I was like, why, why do I think this would help? There's a particular type of report that's used. There's like four or five permutations of this report, at least, that need to calculate, like there's, how do I describe this? There's the way we represent the data in the, in its raw form. And then there's like a translation layer that.
[00:34:55] takes a subset of scores and then depending on the, either the raw scores or the sum of those scores, it categorizes things and just doing that means within Elixir, like we're immutable. So we were copying this, you know, struck in a bunch of different places as it goes through all the different functions to get this data.
[00:35:19] Whereas, uh, I think in an alternate universe we could, each of those functions could just pull the data that they need so that we're not passing around this, you know, mega object of, of stuff. Uh, so that's one way I think that it could improve things. I think, uh, tools like event sourcing and commanded, uh, and even these tables are not the only tools that we have access to.
[00:35:42] Within the database we also have things like triggers and stored procedures that could accomplish some of this. Um, I think there's, there's, I think a healthy debate about whether that's a good approach because it can be very efficient and performant, but it's also a lot of times outside of the source control of your app.
[00:36:03] So if something breaks, how do you roll it back? You can kind of probably do that with migrations and, you
[00:36:09] Dan: We usually manage our triggers and views through migrations, but it does, whenever you have to iterate on them, it is more complicated than just writing an ecto migration. Cause you, you know, especially with certain things, you kind of have to know what's there. Um, and like changing, there's a lot of rules around changing views in Postgres.
[00:36:26] You know, you can't, you can't, you kind of can't change the columns that are there in the order that they're in, but you can always add to the end. Um, you know, if you want to do like a replace, um,
[00:36:37] You know, and then if you need, if you want to drop it, then you have to think, what's the implication on my live running system?
[00:36:42] Um, so, so those things are just kind of pieces to keep in mind. Um, we've moved to using triggers very specifically for, um, just like very specific calculations that, like, We don't, you know, we don't anticipate to change or, or, or you know, just like, um, I think like geospatial is probably the place where this comes in the most of like, okay, well if I write the lat and the long to the two different columns, then like I don't need my code to actually care how to know to make the geo point in geo in, um, in post GIS because I can make a trigger that will, anytime I touch those two columns, update the other one.
[00:37:15] Um. And that makes it so that you can really decouple some of that code. And then you can still, the column can still be there and you can query against it and it can be a virtual column in, in your ectoschema so that you can still load it and read it. Um, but you don't have to necessarily know what it means to then still have it be useful to query against.
[00:37:33] Um, so that can be kind of handy in that regard.
[00:37:36] Owen: Right. Okay. So we have, I think we're deep in the weeds. We haven't quite gotten to the write ahead log of Postgres. I was looking at a database earlier today that was like, Oh, you don't need a server. You can just do everything in the database and we'll broadcast. Like, your database is your application.
[00:37:56] So, everything that was old is new again,
[00:37:59] Dan: Oh, oh, I don't know. Maybe.
[00:38:02] I have like an old man shouting at a cloud kind of feeling about those sentences, but you know, whatever.
[00:38:07] Owen: I will send you some links later
[00:38:09] because I, it's, it was like intriguing. I don't, you know, it's not like an Elixir thing we would do,
[00:38:14] but, uh, I have heard of folks using right head log to kind of get, you know, pub sub stuff. Uh, and I think maybe there's a way of using that here as well, but anyway, so we've talked about structuring the data, querying the data.
[00:38:29] We, we, I think touched on a little bit, like, we, we Not everything happens in SQL. Some of this calculation needs to happen in the code base because,
[00:38:38] just because of the way
[00:38:39] data is structured and reasons, decisions were made. Um, so typically when we're rendering this data, most of the reports that I've been working on have been just straight up HTML tables, uh, with some, uh, with some styling, so CSS, Telman stuff.
[00:38:59] Uh, this application hasn't yet. Requested any kind of charts or graphs. Have you, uh, had to interact with any kind of charts and graphs reports?
[00:39:11] Dan: Yeah, a few. Um, I mean, in some regard, there's just, there's good Well, at least if you're talking about a browser side, there's like a lot of good packages for things like that. And so then it's really just a matter of feeding those tools, the, the JSON scheme or whatever they need to correctly draw, you know, pie charts of certain widths and bar charts of certain heights and things like that.
[00:39:32] Um, and an advantage of some of that, depending on this, the use case, um, we have one where there's a lot of like charts on the homepage that are just, you know, here's what has happened. And, you know, what has happened can be, You know, good as of this morning. So you end up with a very cachable, uh, result, which is, you know, okay, do all the math to figure out what chart I want to render, and then just like cache that big JSON string that you're going to feed to the chart.
[00:39:58] Um, and then now, now you have a fast lookup of the, of the data and the chart can render and everybody can be happy. Um, of course that only works if you know all the inputs and the output is static enough. Um, I haven't had to do a whole lot of charting into PDF, and I think I'm probably just happy about that.
[00:40:16] And I might just leave it at that statement, is I'm happy that I haven't had to draw a lot of charts into PDFs.
[00:40:22] Owen: Right. Yeah. If you're, if you're doing charts and JavaScript and then you're like, Oh, we need to put this in PDFs. That's a yikes.
[00:40:31] Dan: Yeah.
[00:40:32] Owen: Like I know that in Phoenix land, of course, anywhere you can generate SVGs. I know there are some tools I haven't used. I don't think I've used any of those charting tools yet to generate SVG charts.
[00:40:45] Uh, you can absolutely hand roll some SVGs, but SVGs are a bizarro world, because like your, your graph starts in a weird place, and uh, yeah, that's, it's fun. So,
[00:40:59] I
[00:40:59] think uh, if you're generating charts that need to eventually land in a PDF, you probably want to start with SVGs. Because those will show up on the webpage just like they will in a PDF. Hopefully. Hm,
[00:41:12] Dan: with this is like the end of this conversation, right? And like my experience versus your experience versus the team's experience versus the things that are available out in the world. And it, it just strikes me again of like, like pick the tools that your team is passionate about, that your team is good at, that the team like understands and can, you know, rationalize about.
[00:41:27] There's lots of solutions to these problems. Um, you know, and then, And don't fret about it too much, because like, the thing that's gonna bite you is not the thing you think it's gonna be. It's gonna be something else. Uh, I think that's like, been universally true in my experience, especially around reporting.
[00:41:43] Um, and so like, You know, on some degree, like this is where having like import, we talked, we touched a little bit on monitoring as reporting is not what we meant. And it's not what we mean when we talk about building your own reports, like don't build your own monitoring system. There's good ones out there, but use one so that you know where things are falling over.
[00:41:59] And so you know how things are going to perform. And so you can kind of get ahead of these issues that you're going to have kind of no matter what, um, you know, unless you have some magic ability to anticipate all of your users usage patterns, then like good on you. And you know, your startup is probably going very well.
[00:42:13] Mm
[00:42:15] Owen: And, so that's, yeah, so, yeah, Slack alerts are helpful when it comes to error reporting. Uh, also, you're just having that, pinning that tab open so that it's always there in your face. It's a good practice, I think. Um, I mean, I'm just thinking about tables because I've, I, I think I had a dream, Dan, last night about something I was working on.
[00:42:41] I don't know if it was the background job thing or report. I don't remember exactly what the dream was, but I woke up probably at 2 o'clock in the morning being like, Oh, I knew, I know, I know what I need to do. And I was like, I'm not going to write it down or note it in any way. If it's worthwhile, I'll remember it.
[00:42:57] Now all I remember is that something like that happened.
[00:43:00] Uh,
[00:43:01] Dan: you dream in TRs and TDs, I don't know.
[00:43:03] Owen: right. It very well could have been TRs and TDs cause I I've been through a whole evolution myself on what to make a component and what to not make a component. So, and tables are like forms I think are fairly straightforward to componentize.
[00:43:23] Models like a fun thing to like a fundamental experiment. You know, we've got more tools this year than we did a couple of years ago when it comes to the browser, but tables, so many edge cases.
[00:43:35] Dan: Yeah,
[00:43:36] Owen: when you're talking about a report with 20 columns, it's just going to, you know, it's just going to be big, right?
[00:43:41] Dan: It makes me think about, um We were just talking about that callback mess that you can get yourself into in certain object oriented languages. And I feel like components can be great, but then you kind of get into a nest of exceptions. You know, not error exceptions, but like, Oh, but in this case, we want the column to be this way, or to have this heading, or whatever.
[00:44:06] And it's like, okay, my component was good. And now it has like 18 parameters. And anytime I change it, I break things in other places. Um, and so it's like, you know, it's like componentization is coupling. Uh, and so you just got to think that through, um, and like, yeah, maybe reports I think are a good use case in the sense of they tend to show data the same way every time, um, until they don't.
[00:44:35] Owen: I think the mistake I made there was going maybe just one step too far of making every report table a component.
[00:44:45] Um, so like we do have a generic table component. We even have a separate one for filterable stuff. Uh, and even those are painful because I think, now we're a little bit off topic from reports, but there's, uh, I think there's a, there's a camp of people.
[00:45:03] So I love Tailwind. I will use it on every project
[00:45:06] until I die, maybe, but it, uh,
[00:45:09] there are some people
[00:45:11] who along in your life, Owen.
[00:45:13] Right. We'll see. I think, and I subscribed to this for a while, I think I'm pulling back a little bit. I think there were some folks who were like, you should never put anything in your CSS file.
[00:45:23] Dan: Mm hmm.
[00:45:24] Owen: So I do agree that writing custom classes in the CSS file is probably a bad idea.
[00:45:30] That's like you're kind of not actually using Tailwind, right? If you're making a bunch of custom classes in Tailwind. But for things like tables where you want to have a consistent, you know, padding and color system, uh, for your tables, I think the CSS file is really where it should go. Because we've, what we've landed on is we've got these tables and like, these tables need to generally look the same with some, maybe some overrides.
[00:45:55] But representing that with, like, even with the default, uh, Phoenix table component you get with the new app. And with the flop table, FlopPhoenix provides that, uh, there, you have to put all this stuff into like attributes, lists, and like, you have to pass in this keyword list with all this, all these different fields.
[00:46:17] And it's just, I think that's a direction I'd like to pull back from. So, uh, so yeah, TableTalk we're going to, I think projects in my mind in the future could benefit from kind of like custom base styles. So like, when I talk about the app CSS file, I'm thinking just. Elements are getting some, some overrides.
[00:46:41] So we want our headings, our paragraphs, you know, to follow a certain typography. And then we want our, uh, tables to look a certain way, but we're not going to have like card
[00:46:55] Dan: Mm
[00:46:56] hmm. Mm class name inside of our app CSS. Those are a few, few things I think I'd like to put in there. So, uh, also we got tail 1. 4 coming out eventually, but we're I don't know.
[00:47:11] Owen: Is there anything else to talk about? I mean, we've talked through querying stuff. Are there like distinctions between how you've seen reporting play out in Rails and Elixir?
[00:47:21] Dan: No, I mean, I think at the end of the day, right, like Query data, aggregate data, display data, like, there's not a whole lot of difference there. Um, you know, I mean, I think I'd rather use Oban than most other things these days, and I'd rather not have callbacks than in any other case. So, you know. Score two for Elixir in that regard.
[00:47:44] Um, I think the only thing, the only thing I'm looking at on our list and thinking I still want to just bring up, because like, I just don't have a, there's no solution for it, is the like, data too, data too much for viewport. Uh, you know, and it's like, I think this comes up a lot where it's like, well, we, we want all these columns.
[00:48:01] And it's like, well, if you're in Excel or in HTML, that's fine. And then you try to go to paper and you're like, Well, uh, it is, it is wider than landscape. So what do we do? And I was like, well, uh,
[00:48:12] Owen: We could just make it one pixel font, you know,
[00:48:15] Dan: or just here, you can print this, but it's going to be four pages by four pages for every page. Um, you know, and I think that's just, there's something to keep in mind, you know, and it's, you know, it takes, it takes some coaching with customers. It takes some coaching with clients of like, okay, like, you know, um, and we've actually done this in, uh, in one of the products I'm thinking of where If you pick certain options that will make the table very wide, you just can't generate a PDF.
[00:48:40] And we just say like, you can have the HTML version and then all power to you to turn that into A PDF, or you can have the CSV version, but at some point, if you want all these columns, uh, it's just never gonna fit on a, on a reasonable sheet of paper on. So putting it into a PDF doesn't make a ton of sense.
[00:48:56] Um, and so that's just, you know, something else to keep in mind, the viewport, uh. We're not just talking, oh, on my mobile phone, how far do I have to scroll to the right? Um, but we're talking, you know, sometimes it is literally, I'd like to turn this into paper, please, and what does that mean? So, no advice there other than, there be dragons, so watch out.
[00:49:19] Owen: Just say no to printing large tables. Uh, that's fair. Yeah. Uh, well, great. I, so I guess the last thing, I don't know how else to shoehorn this in here, but you mentioned caching. I know you're in Rails. If you're caching stuff in Rails, you're probably using Redis. Is that right?
[00:49:36] Dan: Or Memcached,
[00:49:37] yeah. Okay. Well, pour one out for Redis.
[00:49:41] Yeah, well.
[00:49:44] Owen: TLDR. It's not really, it's not quite open source anymore.
[00:49:49] Dan: There's a lot of license changings going on, so just, you know, be aware of how it affects your use case and your hosting and how you run it. Use
[00:50:00] Owen: out of the box. All
[00:50:04] Dan: you can just use tools you already have, like your Postgres database and your language itself.
[00:50:08] Owen: right. That's my, like, I think I'm constantly reminded of this. I think if there's like one single selling point for like, if I have 10 seconds to sell someone why to use Elixir versus JavaScript, Rails, anything else, it's fewer external sources, like there's so many things I can do with Elixir and the Beam that would require some external service, uh,
[00:50:30] Dan: We'll have to dig through the back catalog to provide a link, but I know one of our episodes was either on Productionizing Elixir, or is Elixir in a polyglot environment? That was one of the points of like, from selling it to your team, selling it to your DevOps team, your server team. Like, yeah, okay, we can all make all these stats around.
[00:50:51] Oh, look at my server load, you know, reduction or, oh, look, I, I need less servers. So we're saving money. Like, great. But also it's like, well, if you want to just get into an environment, you bring a lot less with you. Um, you know, and I was just talking to some of the members of our team, uh, today about our DevOps approach and how we install code and prepare servers and various things like that. Comparing Capistrano on the Rails side to, uh, how we deploy our Elixir application is, and it's like Capistrano is great. It's like, okay, we're going to tell the server, check out the code, you know, link all these things, move things in place, maybe pre compile some things, run migrations, link stuff, restart, huzzah, you know, and that's, and that's awesome.
[00:51:30] And then our, our Elixir deploys follow a very similar approach of, you know, make a fo make a fo make a folder. And then it's just extract binary from, you know, that was built in continuous integration. Uh, and then, Oh yeah. Restart and run migrations and you're done. Um, and so it's just like, nice that you just, you ship, we ship a binary and we're good and there's just like, it's less dependencies to install on the server.
[00:51:54] Like the server doesn't actually even need Elixir installed. It doesn't need Ruby installed. It doesn't need anything installed. Uh, it has everything it needs in the package and you just run it. Um, so yeah, battery's super included.
[00:52:06] Owen: Well, this was a topic, this was an episode about reports and we managed to talk about SQL Elixir, DevOps, Tailwind, CSS, PDFs, printing, all kinds of stuff. So I
[00:52:20] think I mean, maybe the lesson after however long it's been, is that like your reporting touches everything, right? Like it is, you know, how you structured your data, how you stored your data, how you collected your data, uh, and then how you're going to display it, send it, maintain it, keep it up to date. Um, and so, you know, reporting as a topic does have a tendency to impact A lot more than just like, Oh, this controller needs like some work.
[00:52:46] Dan: Um, so.
[00:52:48] Owen: I don't know about you. My favorite report is CI has passed and I see a bunch of green checks. So that's
[00:52:56] Dan: Excellent.
[00:52:57] Owen: if only they could all always be green. Yeah. Well, cool. I think that we've exhausted this topic about reporting and we will have to come back next week with more Elixir Wizards.
[00:53:09] That was fun, Dan.
[00:53:11] Dan: Good talking to you, Owen.
[00:53:13] Owen: Yeah.
\ No newline at end of file