Comparison of different technologies to setup a knowledge management platform for (engineering) projects
Sitemap — — — Webring — — — Repositories — — — About
Abstract: The goal of this article is to discuss the posibility of building a "fully" integrated and open engineering platflorm to develop large, complex and interconected projects. The platform should be able to deal from physically small systems, such as home installations, to large complex systems, such as advance powerplants. Here, in this article, the different goals and technologies are discussed. A final early stage selection of technologies is presented.
Table of Contents
What are we trying to accomplish?
Project integration
In complex engineering projects, there are many moving parts and different fields play a role. Generally, there are mechanical, electrical/electronic, control/intrumentation, design, civil and documentation tasks that have to take place at the same time, on the same system and they all have some relationship with one another. Traditionally, these different aspects of a project have been worked on separately and the data exchange has taken place in a manual way (manual reviews, dumping data in a DB and sharing it, etc).
My goal would be to integrate all if not most of these disciplines within a single system that is fully integrated. As the reader goes through this article, hopefully, it will become clear why and how this vision may be achived.
Digital twin concept
A somewhat hot topic in the engineering world is the idea of a "digital twin". This term refers to a digitalised model of a structure, machine, system, etc; that is fairly well modeled and its outputs are compared to the real world system. The value that this adds to any project is the ability to test new changes or inputs in a digital system and see what it generates. This is a much cheaper way to test new ideas or inputs compared to their implementation in the real world. Another advantage is to compare the outputs that the real world system is generating with those of the computational model. If discrepancies appear, there may be an issue with the way the digital twin is modelled, and therefore, our understanding of the system behaviour; or that the real world system has an issue somewhere.
The theoretical project I am presenting here aims to create a fully integrated platform to develop projects and easily get a digital twin out of it; regardless of its complexity.
Technical goals
Technological openness
The software solutions shall be based on Libre Software as defined by the FSF and the GNU project. Any GPL compatible or incompatible license as defined here will be considered, however, compatible licenses will be preferred.
The choice of software freedom is at the core of the project. Libre software empowers the users and the companies to control how their project is handled, deployed, used and scaled. It also allows smaller deployments save on costs, as certification is generally not required for the tools. Larger deployments can have the different components certified if they so choose.
A software such as REUSE can be used to ensure the openness and license compatibility of the different components. It can be used as a simple SBOM (Software Bill Of Materials). SPDX identifiers will be used. However, for a more complete SBOM system, a framework such as CycloneDX from OWASP could or should be used. A longer and more complete list of software management tools can be found in the Open Source Tools section of the SPDX.
Flexibility of installation and deployment
The system must be as easy to install and deploy. The different parts that may be needed shall be simple to install in any *NIX environment by themselves and shall be simple to interoperate. They should also be able to use newer deployment methods, such as containers, to allow the users to pick their preferred installation method. Some installation systems that I would like to target are:
- Guix: it is a GNU project used to manage and deploy software independently of the Linux platform in use. It fosuces on flexibility of software choice (versions, patches, environments) and reproducibility.
- General package managers: the packages should be already packaged or be easily packaged on common package managers, such as APT for the Debian family, OpenSUSE's zypper, Fedora's DNF or FreeBSD's and NetBSD's port systems.
- Container technology such as Podman, Apptainer, Sarus, Docker, Kubertenes or Cloud Native Computing Foundation technology.
Centralization vs decentralization
Single engineering applications are normally always installed in the user's computers where they run locally. However, large and complex engineering solutions generally require some IT/Computer knowledge in order to correctly deploy the software. This indicates an issue where, if several engineering tools and related technologies have to interact with each other, local deployments may not be easily achieved. For this reason, the project proposed here would have a fully centralized server, where all the computations, analysis and outputs take place on a centralized system. However, each individual tool can be installed locally by the users, because they should be simple to install and are libre, so they may work locally with data that can be downloaded from the centralized system.
Change traceability
Any serious project will grow in size, people and complexity. Tracking the different changes allows us to better understand the history of the project and give responsability to the users of the system. It also allows mistakes to be rolled back in case they were done accidentally or were issued incorrectly. Some of the most important tools in the software industry are specifically designed for this, such as Git, Fossil, Gitea, Dolt and others.
Integration of information
All information in the system shall be integrated, easily accessible and auditable. Each component shall carry all of its relevant information for its analysis, construction, integration, changes and verification. All information shall be exportable in a clean and ordered manner for its auditability and use in external systems. Updates to the system should be reflected in the documentation. A flagging mechanism should be in place in such a way that if a relevant piece of data has been updated, which affects the documentation in some form, it will indicate the user that a potential update is needed; even if the user is not directly dealing with the part where the data was modified.
Automatic report generation will also be needed. For example, after the design, the user may want to know what parts are connected to another system and show the mechanical stresses that exist within each other. The documentation system shall be able to handle this case and display the relevant data (list of parts and stresses), but also pictures of the stresses if that is requested.
Confidentiality
Large and complex projects are generally jointly carried out by several parties, each party shall only have access to a portion to the information in the project and not more. Within each organisation, different people may have a higher level of clearance than others. These particularities require that the proposed system be able to handle the information that each party receives to preserve the confidentiallity of the project. This must also take place while allowing users to still work on their specific system even if they don't have access to all the information.
Flexibility for the user and third-parties
Each user of the system may have a different set of requirements. This particularity entails that the system will have to deal with an ever changing set of necessities and particularities. To be able to cope with this problem, a plug-in/extension system is generally used.
A plug-in system allows the users of a program to modify and fit the system to their needs, whatever those may be. It may also have advantages with regards to the development workflow, as new ideas could be prototyped quickly using the extension system. Finally, the extension system decouples the core of the application, which needs to be carefully monitored and taken care of, with the tools that a particular/non-standard task may need.
- Integration with external tools
Integration with external tools could be directly dealt with by using the plug-in system. However, some standarised form of communication should be in place. Something like a REST API, GraphQL, or maybe even plain HTTP should be available so that other third-parties can consume data directly from the system. The project could make use of the FMI and FMU standards for this task.
Correctness and integrity of the system
The system shall not break under incorrect changes. Mistakes should be detectable as early as possible withing the development workflow. The system should easily and quickly flag this issues and not allow a user to commit an incorrect change unless forced. Some early sanitation tests may be needed. Further systematic and automated checks, such as CI/CD, should function nicely within the system.
TODO Feasibility and sustainability
Base programming language
Comparison table between programming languages
The following table indicates the most widely used programming languages in the so-called "back-end" server world. The reason to focus into these languages is because the system that is being proposed would need to have a very important server component to it.
Language | Implementation(s) | Licence | Target | Frameworks | Community size | Success | Notes | Typed |
---|---|---|---|---|---|---|---|---|
Elixir | Compiler for BEAM/JIT | Apache v2 | Servers, Web | Phoenix, Nerves, Ash UI: Liveview | Medium | Medium, growing | See1 | Runtime, annotations2 |
Go | Compiler | BSD v3 | Servers, Applications | Several | Very large | Huge, growing | Yes, basic | |
JavaScript | Several | - | Web, Servers | Fastify, Meteor, KoaJS, ExpressJS UI: ReactJS, VueJS | Huge and very dynamic | Enormous, growing | See3 | No, objects |
TypeScript | Compiler to JS or WASM | Apache v2 | Web, Servers | NestJS, AdoniJS, Feathers, Ts.ED, Remix, Qwik UI: Svelte, Angular, NextJS, VueJS/NuxtJS | Very large and growing | Large, growing | Yes | |
PHP | Interpreter/JIT | PHP | Servers | Lavarel, Symfony, Yii, Slim, CakePHP | Very large. Old | Enormous, dwindling | See4 | No, objects |
Python | Interpreter, JIT, Compiler | PSF v2 & BSD0 | Servers, Applications | Django, Flask, Tornado, Sanic | Very large | Huge, growing | Runtime, annotations | |
Ruby | Interpreter/JIT | BSD v2 | Servers | Ruby on Rails, Sinatra, Hanami | Enormous, stable | Runtime | ||
Java | JIT/Compiler | GPLv2 | Applications, Servers | Spring, Hilla, Wicket, Dropwizard, Struts | Huge. Old | Enormous, dwindling | Yes | |
Kotlin | JIT/Compiler, Compiler | GPLv2 | Applications, Servers, UIs | Very large | ||||
Clojure | JIT/Compiler, JS compiler | GPLv2 | Servers, Web, Applications | Medium | ||||
Common Lisp5 | Several | - | Applications, Servers | TODO | Small | Medium, dwindling | It can be, but no | |
Ada6 | Compiler | LGPLv3 | Applications, Servers | AWS | Very small | Low |
TODO General notes about each programming language
The following sections contain oppinionated and potentially inaccurate information and incorrect conclusions.
Java and JVM-based programming languages
Java is a very well-known programming languages. It is most likely in the top 5 most widely used programming languages. However, it shows its age in some of the design decisions (very heavy on object-oriented programming). Running the JVM is also a taxing process. Small SBCs7 generally cannot sustain it with some other processes also running (think of 1GB RAM SBCs). This goes against the platform being resource efficient. It is true that smaller and more resource aware JVM configurations are used (Android is a wonderful example of this), but that would require manual tweaking from the user side or more work from our side. Also, portability of the JVM to new architectures or systems is a difficult task due to the JVM's size and complexity. Bootstrapping the JVM is also a problem as it is implemented in JAVA.
- Kotlin
Kotlin aims to be an improved, mordern version of Java. Google now recommends it for Android development and modern tools are designed for it. It tries to be compatible in nature with Java. There is also a native Kotlin compiler based on the LLVM toolchain.
- Clojure
It is a modern, batteries included LISP. It can also be transpiled to JavaScript easily. It is desinged to use the power of the JVM while taking full advantage of LISP's nature. It is probably the most widely used LISP in the "professional" world.
JavaScript and its chaotic world
JavaScript and more recently TypeScript have grown tremendously in the past years due to the prevalence of web technologies outside of the normal "web browsers". Modern applications, both in mobile and desktop are now using JS. Server backends are also being deployed with JS/TS thanks to Node. This allows for the entire technology stack to be based on the same language. JS/TS in the backend, with tons of libraries to aid in database management, dependency management, communications with the clients and other services. JS/TS in the frontend, making it portable, dynamic, loaded with CSS and HTML.
TODO PHP and its ancient yet alive world
TODO Python, good for everything
TODO Ruby and Rails
TODO Elixir and the BEAM
TODO Go and the backend infraestructure world
Conclusion
- Java: Kotlin seems to be picking up steam and it seems to be set
to overtake it. While Java keeps improving and has one of the
largest sets of libraries and programmers, and the enterprise world
seems to love it, its time seems to have passed. Oracle, the main
developer of Java, also has some shady history and it seems
untrustworthy, however, their behaviour should not impact the Java
ecosystem.
- Result: NO PASS
- Kotlin: modern Java. However it still requires the use of the JVM
and the native compiler cannot make use of the JVM tools and
libraries. There is also Jetpack Compose and it web counterpart,
Compose-Web, which would allow developing fronteds with the same
tools as Android and desktop, but for the web. This allows using
Kotlin + Jetpack as the two main tools for the technology
stack. Another issue is that Kotlin has still not been bootstrapped,
so a binary has to be used.
- Result: KEEP AN EYE ON IT. But for the time being, NO PASS
- Clojure: loved by the LISP world. Has some great people behind
it and it can take full advantage of the JVM stack. With the use
of ClojureScript, it can also be used for the frontend, which
greatly reduces the technology stack. However, the same negative
comments regarding the JVM still stand.
- Result: KEEP AN EYE ON IT. But for the time being, NO PASS
- JavaScript: no. Use TypeScript.
- Result: NO PASS
- TypeScript: TODO
- Result:
- PHP:
- Result:
- Ruby:
- Result:
- Python:
- Result:
- Go
- Result:
Ranking
Language | Result | Position |
---|---|---|
Elixir | 1 | |
Go | 1 | |
TypeScript | 2 | |
Kotlin | KEEP AN EYE ON IT. | 3 |
Clojure | 3 | |
Ruby | 4 | |
Python | 4 | |
PHP | 5 | |
JavaScript | NO PASS. Use TypeScript | 5 |
Java | NO PASS. Use Kotlin | 6 |
Database for knowledge management
Introduction to the conceptual design
In order to manage all the information that is going to be taken into account for a large project, the choice of a database is of upmost importance. As of the writing of this text, on the
, here is what I imagine the data structure would look like:First things first, how are the different components related to each other? This provides a high level overview of what an actual system would look like:
- And each component would have the following information:
Ideally, there would also be a second nearly identical copy of this data that would show real-world measurements, in real-time hopefully. That way users would be able to see what the real world system is generating and compare it directly to what the digital twin produces.
Comparison table for DB technologies
Note, the success column is subjective and the number of stars was taken mostly on the
.Database | License | Language | Type | Drivers | Success | Use | Package presence | Notes |
---|---|---|---|---|---|---|---|---|
ArangoDB | Apache v2 | C++ | Graph, Document, Search | Java, JS, Go, C#, community | 12.8k stars | |||
BadgerDB | Apache v2 | Go | NoSQL, Key-Value | Go | 11.7k stars | DGraph | Alternative to RocksDB, BoltDB. Embeddable | |
Cockroach | BSL8 | Go | Document, SQL | Go, JS/TS, Python, Java, Ruby, C, C#, Rust | 26.4k stars | Industrial | Kubertenes, Docker9, Manual10 | Focus on replication |
Consul | MPL v2 | Go | Distributed NoSQl | HTTP API | 25.9k stars | |||
CouchDB | Apache v2 | Erlang | Document | Java, Erlang, JS, Ruby, Python | 5.6k stars | |||
DGraph | Apache v2 | Go | Graph | Go, Java, JS, Python, C#, gRPC | 18.9k stars | |||
DuckDB | MIT | C++ | Data analytics | |||||
EdgeDB | Apache v2 | Python | Graph-Relational | Rust, C#, JS, Python, Go | 10.2k stars | Based on PostgreSQL | ||
etcd | Apache v2 | Go | Distributed Key-Value | Go, gRPC | 42.4k stars | |||
FerretDB | Apache v2 | Go | Document | Plenty11 | 10/1011 | Industrial | Alternative to MongoDB | |
InfluxDB | MIT | Go, Rust12 | Timeseries | JS, Python, Go, Java, PHP, community | 24.8k stars | The company has had issues with customers, it also seems to be difficult to build | ||
KeyDB | BSDv3 | C++ | Key-Value13 | Redis wire protocol | 6.7k stars | |||
Manticoresearch | GPLv2 | C++ | Search | MySQL wire protocol | 2.4k stars | Alternative to Elasticsearch | ||
MariaDB | GPLv2 | C | SQL, Graph14, NoSQL15? | Plenty | 10/10 | Industrial | Everywhere | NoSQL requires BSL extension |
Memcached | BSDv3 | C | Key-Value, Cache | Plenty | 12.4k stars | Industrial | Everywhere | Mostly only used as cache |
Milvus | Apache v2 | Go, C++ | Vector(?) | Python, Java, Go, C++, JS | 14.7 stars | |||
MindsDB | GPLv3 | Python | DB-AI | The DB of choice | 13.1k stars | DB data analysis | Not a real DB | |
NebulaGraph | Apache v2 | C++ | Distributed Graph | Java, Python, C++, Go, Node | 8.5k stars | |||
Neon | Apache v2 | Rust, Python | Serverless PostgreSQL | |||||
OpenObserve | Apache v2 | Rust, Vue | ElasticSearch alternative | ElasticSearch API | ||||
PostgreSQL | MIT-like | C | SQL, Graph16, NoSQL, Timeseries17, Distributed18 | Plenty | 10/10 | Industrial | Everywhere | |
Redis | BSDv3 | C | Key-Value, Document | Plenty | 10/10 | Industrial, Cache | Everywhere | Advanced plugins are not libre |
RethinkDB | Apache v2 | C++ | Realtime | JS, Python, Ruby, Java and community | 25.9k stars | |||
Riak | Apache v2 | Erlang | Key-Value, Timeseries, S3 | |||||
RocksDB | GPLv2 or Apache v2 | C++ | Key-Value | C++, Java | 24.4k stars | |||
Rqlite | MIT | Go | Distributed SQL(ite) | SQLite | 13k stars | |||
ScyllaDB | AGPL-3.0 | C++ | Distributed (Apache Cassandra compatible) | Same as Cassandra | TODO | |||
SQLite | CC-0 | C | SQL, Document | Plenty | 10/10 | Industrial, Simple | Everywhere | Embeddable and Safety-critical |
TerminusDB | Apache v2 | Prolog/Rust | Graph | TODO | ||||
TDengine | AGPLv3 | C/C++ | Timeseries (IoT) | Java, Go, Python, JS, C#, Rust | 20.5k stars | |||
TiDB | Apache v2 | Go | Distributed SQL | MySQL wire protocol | 33.2k stars | |||
Typsesense | GPLv3 | C++ | Search | JS, PHP, Python, Ruby, community | 12.1k stars | Alternative to Elasticsearch, Algolia | ||
VictoriaMetrics | Apache v2 | Go | Timeseries | TODO | ||||
Weaviate | BSDv3 | Go | Vector | TODO | ||||
WhiteDB | GLPv3 | C | TODO | |||||
Vitess | Apache v2 | Go | Distributed SQL | MySQL wire protocol | 15.4k stars | |||
YugabyteDB | Apache v2 | C++ | SQL | PosgreSQL wire protocol | 7.4k stars | Goal: distributed PostgreSQL | ||
Dolt | Apache v2 | Go | Git for data | - | 14k stars | |||
ImmuDB | Apache v2 | Go | Immutable DB, Key-Value, SQL | Java, Go, C#, Python, JS | 8k stars | Tracks changes to the DB | ||
Irmin | ISC | Ocaml | Distributed | |||||
Litestream | Apache v2 | Go | Stream SQLite changes to storage | - | 8k stars | Make SQLite resilient |
Platform integration technologies
Something that may more closely bring the entire infrastrusture together is a Backend-as-a-Service solution. However, I have not taken a closer look into this possibility, as these BaaS seems to be modelled for more traditional web services in mind.
Platform | License | Language | DBs | Auth | Success | Deployment | Notes |
---|---|---|---|---|---|---|---|
Appwrite | BSDv3 | TypeScript, PHP | MariaDB, Redis (GraphQL) | 28.7k stars | |||
Nhost | MIT | TypeScript | PostgreSQL, Hasura/GraphQL | Hasura | 5.8k stars | GraphQL integration | |
PocketBase | MIT | Go, (Svelte) | SQLite (WAL) | OAuth v2 | 20.3k stars | Very simple | Small embeddable and single server |
Supabase | Apache v2 | TypeScript (React), Elixir | PostgreSQL (GraphQL) | 44.4k stars |
Anciliary technologies
In order to carry out this grand vision, we need to make use of some of the best tools that exist in the libre world. Here are the following topics that need to be covered and their respective technological solution.
Visualisation
VTK and/or the Open Cascade (OCCT) geometry kernel, shall be used for the visualisation of the system. VTK offers different solutions: native, web (server side) and web (client side). Sidenote, for HPC system and modern (threaded, parallel) visualization there is the new VTK-m toolkit. Both technologies can be used together for different things; VTK for data and mesh visualization and Open Cascade (OCCT) for the geometrical descriptions. Projects such as Salome follow this approach.
License
VTK is licensed under the BSDv3 license. Open Cascade (OCCT) is licensed under the LGPLv2.1 with the Open CASCADE linking exception. VTK-m is licensed under the BSDv3.
System simulation
OpenModelica is a system simulation tool based on the Modelica programming language. It can be used to simulate electrical, mechanical, thermal, hydraulic, control systems among others. It can also couple them together.
Each component on the system could have its own Modelica equivalent model which could describe its behaviour. A complete interconected Modelica model should be auto-generated from the information in the model.
For example, lets assume we have the following linear system:
- Reservoir (ground level)
- Pipe
- Pump
- Pipe
Reservoir (higher level)
The system should be able to generate each element's Modelica block, connect it with its neighbours and correctly setup the values to generate a correct simulation, such as the different pressure losses that take place from the suction and discharge sides.
Note: it may be worth looking into CMB and SMTK as the simulation writers, drivers and couplers between programs/systems.
TODO Modelica libraries
The Modelica world is thriving and the community has already developped a vast ammount of libraries for a wide range of problems. One major issue is that most libraries use their own built in solutions in order to develop their behavior. One such example is that many thermal libraries all develop their own fluids module and library. This generally causes them to be incompatible with each other. The following libraries may be worth looking into in order to lighten up the development time.
License
OpenModelica is licensed under the GPLv3 or the OpenModelica License.
Major dependencies
- Language: C++
- Tools
- Compiler:
- LLVM/Clang. License: Apache v2
- Qt (for the GUI). License: LGPLv3 with some modules under the GPLv3
- Modelica Standard Library. License: Modelica license19
- Compiler:
TODO Mechanical, structural and thermal simulations
TODO Fluid simulations
Documentation and plans
There shall be an automatic documentation system that takes the information from the project and translates it into printable paper. However, there are several ways this needs to be done with several alternatives.
Technical documentation
Technical documentation refers to P&IDs, control diagrams, electrical systems, electronic devices, etc. For each one of them there exists a Libre package able to deal with them.
- Control diagrams, basic P&IDs and electrical systems
QElectroTech is a diagraming tool that can create control, electrical, hydraulic and thermal systems in a "flat" design and follows the IEC 60617. It can export designs as PDF or DXF. As of version 0.9 it does not seem to be able to create links to labels in other sheets.
For electronic diagrams KiCAD could also be used. It integrates nowdays SPICE capabilities, otherwise Qucs-S could be used for simulations.
- License
QElectroTech is licensed under the GPLv2 license. KiCAD is licensed under the GPLv3 license. Qucs-S is licensed under the GPLv2 license.
- License
- Blueprints, isometrics
FreeCAD is the only 3D CAD program that I am aware as of that can create 2D drawings starting from a 3D design. However, quite a bit of work would need to go into cooperating the system with FreeCAD.
- License
FreeCAD is licensed under GPLv2 license.
- License
- Diagrams
Text documentation
Text documentation refers to readable prose with potentialy some data intertwined in it. Here are the potential systems that could generate such documents.
- LaTeX
LaTeX is the standard documentation format for high quality and academic works. The documents are written in plain text and then compiled to a working PDF. It has a lot of packages to create fancy structures, such as multipage tables. However, it requires the installation of its complete infraestructure. Ideally, LuaLaTeX should be used as it provides a more modern typesetting system with programmability in mind. One project that exploits LuaLaTeX's programmability is SpeedData, which uses XML to automatically output beautifully typeset documents.
- License
LaTeX is licensed under the LaTeX Public License (LPPL).
- License
- Pandoc
Pandoc is a document translator system. It takes a document from one format, and outputs it in another. This may not seem useful as a document system. However, the information within the system could be generated in a simple format, such as Markdown or reStructuredText and then process it into a final document. However, the generation of PDFs requires the use of LaTeX20.
Sadly, if Pandoc is built from source, it requires a gigantic array of Haskell modules and the GHC.
- License
Pandoc is licensed under the GPLv2-or-later.
- License
- LibreOffice
LibreOffice is a libre alternative to the M$ Office suite. It can be used without the GUI (Graphical User Interface) using the
--headless
option, which allows it to be used through its API. Final PDFs could be generated by creating a document writter either directly or using LibreOffice Base.- License
LibreOffice is licensed under the GPLv3, LGPLv3 and MPLv2.
- License
TODO Job management
The platform will require a job manager to control what analysis, simulation, tasks, etc are carried out. The job manager has the task to control the resources and notify when the job is done (among other secondary tasks). One technology that is commonly used in HPC to manage jobs is SLURM.
License
TODO Deployment management
Deploying such a large and complex system requires some help to get it right. For this reason tools such as Kubernetes (K3S), Rancher, Warewulf, Cockpit may come in handy even if they are only used for internal development.
TODO Licenses
Monitoring
Whatever the final implementation of the system is, it will require a level of performance monitoring. The actual project, such as simulations or real-time data, will also benefit from a built-in monitoring system. Two solutions are provided: Prometheus is a monitoring system and a real-time database. It can be used alone or it can also be used with the other potential solution, Grafana, an observability and visualisation platform. Grafana can agregate data from other sources, such as Loki, InfluxDB, PostgreSQL among others. Netdata is another very well-known solution.
License
Prometheus is licensed under the Apache v2 license. Grafana is licensed under the AGPLv3 license. Netdata is licensed under the GPLv3 license.
TODO Authentification and permission management
TODO Continious Integration, Quality Management
CI/CD…
To begin with, a prototype
Okay, after all these topics and discussions, I have reached some conclusions, but a lot more needs to be done than just write a few comparison matrixes and descriptions. Any project needs to start somewhere, a prototype needs to be created.
The technology used in any prototype should resemble the one in the final product, but it does not have to be the exact same. It is also useful to better understand the problem and the technical challenges associated with it. Even more, for my personal case, it will also serve as a training ground for me to learn a large number of technologies that I will have to use in the future. For that reason, here is the technology stack that I have chosen to begin with.
- Platform/Database: PocketBase/SQLite This may seem like a stupid choice to begin with. However, it is quite intentional. Pocketbase provides a simple to use backend… Actually, it is too simple, but that is the point. It should allow for quick and easy testing of ideas. It uses SQLite behind the scenes, which for a simple prototype will be more than enough.
- Visualization: OpenCascadeJS (OC.js) (an example of its use would be CascadeStudio). It is a "simple" JS/TS library that allows loading OpenCascade models and operations into the browser.
- Frontend programming language: Vue 3 and TypeScript. As the Kitware stack makes use of Vue 3, it will be the JS/TS framework of choice. Some examples that consume Vue code are Trame, Glances and VolView. Vue also seems to be one of the fastest and largest JS/TS frameworks (source).
Backend programming language: Go. As PocketBase is written in it, it will be most helpful and make most sense to utilise the same technology that the selected backend uses.
A few more notes. The technology selection above does not necessarily represent the final technology stack. Here are a few examples:
- Database: a large system will require a more powerful and flexible DB infraestructure. DB systems such as PostgreSQL (with its different extensions) and MariaDB may be better suited for the final task. However, I would still like to support SQLite as it is simple, small but also quite capable. It may also come in handy during development and testing.
- Backend programming language: while Go is a perfectly fine language for backends and it is already used in a ton of backend infraestructure, the Elixir programming language seems to focus quite nicely on reliability and resiliance. This is a very desirable property for the final product (plus it should be getting type support starting with v1.16).
- Frontend programming language: the prototype will be based on web technologies. However, a native app for the final system may be more desirable. Maybe a platform like Salome could be leveraged to run the final user facing application, which would require the use of C++ and Python in this case.
- Mentioned as part of the goals of the project, the system should be able to run as a container(s). However, one interesting thing that may be worth looking into, is the use of Unikernels21 for enhanced security, performance and role separation.
TODO Add/assess technologies
Fluent-bit [https://fluentbit.io/] Fluentd [https://www.fluentd.org/] Earthly [https://earthly.dev/] Aerospike [https://github.com/aerospike/aerospike-server] DevPod [https://devpod.sh/] Fuzzball (not open!) [https://ciq.com/products/fuzzball/]
Footnotes:
However, José Valim, the creator of Elixir has expressed his wish to push typed Elixir.
Eventhough JavaScript is a rapidly growing (and already huge) ecosystem, it has quite a few notable drawbacks.
- The technology and frameworks are constantly changing. This means, that after a few years, the community may move towards a newer, better maintained and fresher system. This leaves the older ecosystem to very slowly fade away. This also indicates that there is a lot of market fragmentation. A lot of frameworks have designed the same tools for themselves.
- The quality of most packages in the JavaScript/Node ecosystem is low, though there are really good ones!
- JavaScript lacks types, which does make the code more brittle as it grows. This is why TypeScript was developed by Microsoft.
The main interpreter implementation is improving performance,
safety and adding newer features. However, PHP does show its age, see
$
for variable indentifiers. Most new projects do not use it, though
it has some very well-known and powerful frameworks.
The addition of Common Lisp, from now on CL, may seem like a strange and out of touch addition, as it is a language that basically no body uses in large complex systems. However, since this project requires the interoperation of so many tools and data systems, CL does provide a large advantage as it can be used as a meta-programming language for all. This is, in the end, the biggest strength of CL.
Just like with Common Lisp, Ada seems an odd choice. However, its extreme focus on quality, maintainability and readability do check the most important properties in a robust system. Therefore, it is added to the list for completeness.
Single Board Computers.
3 year period. Acceptable.
Docker is not recommended.
Manual build is based on Bazel, a Java based build system. It also requires a C++ compiler.
MongoDB "drop-in" replacement .
In the future, it is expected that InfluxDBiox, written in
Rust, will take over the core of InfluxDB. InfluxDBiox, aka InfluxDB
3, has already launched an it is based on Rust and Apache Arrow.
It is a multithreaded fork of Redis.
The Modelica license is an open license that does not allow the selling of the licensed product by itself. It can only be sold when packaged with a value added asset.
This is not entirely true. Other systems can be used. However, LaTeX is the most common one and it generally creates the most consisten output.