Surreal Engine C++ WebAssembly Port

In this note, I will describe how I ported the Surreal Engine game engine to WebAssembly.

Surreal Engine is a game engine that implements most of the functionality of the Unreal Engine 1. Well-known games on this engine include Unreal Tournament 99, Unreal, Deus Ex, and Undying. It is a classic engine that primarily worked in a single-threaded execution environment.

Initially, I had the idea to take on a project that I would not be able to complete in any reasonable timeframe, thus showing my Twitch followers that there are projects that even I cannot accomplish. On the very first stream, I suddenly realized that porting the Surreal Engine C++ to WebAssembly using Emscripten was achievable.

Surreal Engine Emscripten Demo

A month later, I can demonstrate my fork and build of the engine on WebAssembly:
https://demensdeum.com/demos/SurrealEngine/

The controls, as in the original, are carried out on the keyboard arrows. Next, I plan to adapt for mobile control (touch), add correct lighting, and other graphical features of the Unreal Tournament 99 renderer.

Where to Start?

The first thing I want to say is that any project can be ported from C++ to WebAssembly using Emscripten; the question is only how complete the functionality will be. Choose a project whose library ports are already available for Emscripten. In the case of Surreal Engine, it was very fortunate because the engine uses the SDL 2 and OpenAL libraries, both of which are ported to Emscripten. However, the graphical API used is Vulkan, which is currently not available for HTML5. Work is underway to implement WebGPU, but it is also in draft stage, and it is unknown how simple the further port from Vulkan to WebGPU will be after its full standardization. Therefore, I had to write my own basic OpenGL ES / WebGL renderer for Surreal Engine.

Project Build

The build system in Surreal Engine is CMake, which also simplifies porting since Emscripten provides its native builders – emcmake, emmake.
The Surreal Engine port was based on the code of my latest game on WebGL/OpenGL ES and C++ called Death-Mask, which made the development much easier; all necessary build flags and code examples were with me.

One of the most important moments in CMakeLists.txt is the build flags for Emscripten. Below is an example from the project file:

set(CMAKE_CXX_FLAGS "-s MIN_WEBGL_VERSION=2 \
-s MAX_WEBGL_VERSION=2 \
-s EXCEPTION_DEBUG \
-fexceptions \
--preload-file UnrealTournament/ \
--preload-file SurrealEngine.pk3 \
--bind \
--use-preload-plugins \
-Wall \
-Wextra \
-Werror=return-type \
-s USE_SDL=2 \
-s ASSERTIONS=1 \
-w \
-g4 \
-s DISABLE_EXCEPTION_CATCHING=0 \
-O3 \
--no-heap-copy \
-s ALLOW_MEMORY_GROWTH=1 \
-s EXIT_RUNTIME=1")

The build script itself:

clear
emmake make -j 16
cp SurrealEngine.data /srv/http/SurrealEngine/SurrealEngine.data
cp SurrealEngine.js /srv/http/SurrealEngine/SurrealEngine.js
cp SurrealEngine.wasm /srv/http/SurrealEngine/SurrealEngine.wasm
cp ../buildScripts/Emscripten/index.html /srv/http/SurrealEngine/index.html
cp ../buildScripts/Emscripten/background.png /srv/http/SurrealEngine/background.png

Next, prepare index.html, which includes the project file system preloader. For web deployment, I used the Unreal Tournament Demo version 338. As can be seen from the CMake file, the unpacked game folder was added to the build directory and linked as a preload-file for Emscripten.

Main Code Changes

Then it was necessary to change the game’s main loop; launching an infinite loop is not allowed as it causes the browser to hang. Instead, you need to use emscripten_set_main_loop. I wrote about this feature in my 2017 note “Porting an SDL C++ game to HTML5 (Emscripten)
Change the while loop exit condition to if, then output the main game engine class that contains the game loop to the global scope, and write a global function that will call the game loop step from the global object:

#if __EMSCRIPTEN__
#include 
Engine *EMSCRIPTEN_GLOBAL_GAME_ENGINE = nullptr;
void emscripten_game_loop_step() {
	EMSCRIPTEN_GLOBAL_GAME_ENGINE->Run();
}
#endif

After this, make sure that there are no background threads in the application. If there are, be prepared to rewrite them for single-threaded execution or use the pthread library in Emscripten.
A background thread in Surreal Engine is used for music playback. From the main engine thread, data about the current track and the need to play or stop music is received. The background thread then gets the new state via mutex and starts playing new music or pauses. The background thread is also used for music buffering during playback.
My attempts to build Surreal Engine under Emscripten with pthread were unsuccessful because the SDL2 and OpenAL ports were built without pthread support, and I didn’t want to rebuild them for the sake of music. Therefore, I transferred the background music thread functionality to single-threaded execution using a loop. By removing pthread calls from the C++ code, I moved buffering and music playback to the main thread, increasing the buffer by a few seconds to avoid delays.

Next, I will describe the specific implementations of graphics and sound.

Vulkan is Not Supported!

Yes, Vulkan is not supported in HTML5, although all advertising brochures present cross-platform and wide support on platforms as the main advantage of Vulkan. For this reason, I had to write my own basic graphics renderer for the simplified type of OpenGL – ES. It is used on mobile devices, sometimes lacking modern OpenGL features, but it is very well portable to WebGL, which is implemented by Emscripten. Writing the basic renderer for tiles, bsp rendering, simple GUI display, and models + maps took two weeks. This was perhaps the most challenging part of the project. There is still a lot of work ahead to implement the full rendering functionality of Surreal Engine, so any help from readers in the form of code and pull requests is welcome.

OpenAL is Supported!

It was very fortunate that Surreal Engine uses OpenAL for sound output. Writing a simple hello world in OpenAL and building it on WebAssembly with Emscripten made it clear how simple everything is, and I set off to port the sound.
After a few hours of debugging, it became apparent that there are several bugs in the Emscripten OpenAL implementation. For example, when initializing to read the number of mono channels, the method returned an infinite number, and after attempting to initialize a vector of infinite size, C++ crashed with a vector::length_error exception.
This was bypassed by hardcoding the number of mono channels to 2048:

		alcGetIntegerv(alDevice, ALC_MONO_SOURCES, 1, &monoSources);
		alcGetIntegerv(alDevice, ALC_STEREO_SOURCES, 1, &stereoSources);
#if EMSCRIPTEN
monoSources = 2048; // for some reason Emscripten's OpenAL gives infinite monoSources count, bug?
#endif

Is there a Network?

Surreal Engine currently does not support network play. Bot play is supported, but someone needs to write AI for these bots. Theoretically, network play on WebAssembly/Emscripten can be implemented using Websockets.

Conclusion

In conclusion, I want to say that porting Surreal Engine was quite smooth due to the use of libraries that have Emscripten ports, as well as my previous experience in implementing a C++ game for WebAssembly on Emscripten. Below are links to sources of knowledge and repositories on the topic.
M-M-M-MONSTER KILL!

If you want to help the project, preferably with WebGL/OpenGL ES renderer code, write to me on Telegram:
https://t.me/demenscave

Links

https://demensdeum.com/demos/SurrealEngine/
https://github.com/demensdeum/SurrealEngine-Emscripten
https://github.com/dpjudas/SurrealEngine

Flash Forever – Interceptor 2021

Recently, it turned out that Adobe Flash works quite stably under Wine. During a 4-hour stream, I made the game Interceptor 2021, which is a sequel to the game Interceptor 2020, written for the ZX Spectrum.

For those who are not in the know – the Flash technology provided interactivity on the web from 2000 to around 2015. Its shutdown was prompted by an open letter from Steve Jobs, in which he wrote that Flash should be consigned to history because it lagged on the iPhone. Since then, JS has become even more sluggish than Flash, and Flash itself has been wrapped in JS, making it possible to run it on anything thanks to the Ruffle player.

You can play it here:
https://demensdeum.com/demos/Interceptor2021

Video:
https://www.youtube.com/watch?v=-3b5PkBvHQk

Source code:
https://github.com/demensdeum/Interceptor-2021

CRUD repository

In this note, I will describe the basic principles of the famous classic Crud pattern, the implementation in Swift. Swift is an open, cross -platform language available for Windows, Linux, MacOS, iOS, Android.

There are many decisions of abstracting the storage of data and the logic of the application.One of these solutions is the CRUD approach, this is acronym from C – Create, R -read, U – Update, D – Delete.
Typically, the implementation of this principle is ensured by implementing the interface to the database in which work with elements occurs using a unique identifier, such as ID.An interface is created for each letter Crud – Create (Object, ID), Read (ID), Update (Object, ID), Delete (Object, ID).
If the object contains ID inside itself, then the ID argument can be missed in terms of the methods (Create, Update, Delete), since the object is transmitted there entirely with its field – ID.But for – Read requires ID, as we want to get an object from the database by the identifier.

All names are fictitious

Imagine that the Assistantai hypothetical application was created using an Etherrelm free SDK database, the integration was simple, the API was very convenient, as a result, the application was released into markets.
Suddenly, the SDK Etherrelm developer decides to make it paid, setting the price of $ 100 per year per one user of the application.
What?Yes!What now to do to developers from Assistantai, because they already have 1 million active users!Pay $ 100 million?
Instead, it is decided to evaluate the transfer of the application to the Rootdata database, according to programmers, such a transfer will take about six months, this without taking into account the implementation of new features in the application.After short thoughts, it is decided to remove the application from the markets, rewrite it on another free cross -platform framework with a Buems built -in database, this will solve the problem with the database + will simplify the development for other platforms.
A year later, the application is rewritten to Buems, but then suddenly the developer of the framework decides to make it paid.It turns out that the team got into the same trap twice, whether they will manage to get out for the second time, this is a completely different story.

Abstraction to help

These problems would be avoided if the developers used the abstraction of interfaces inside the application.For three whales of OOP – polymorphism, incapsulation, inheritance, not so long ago added one more abstraction.
The abstraction of the data allows you to describe ideas, models in general terms, with a minimum of details, while quite accurate for the implementation of specific implementations that are used to solve business abomes.
How can we abstract the work with the database so that the logic of the application does not depend on it?We use CRUD!

Simplified UML CRUD scheme looks like this:

An example with a fictional data database:

An example with a real SQLite database:

As you have already noticed, when you switch the database, only it changes, the CRUD interface with which the application interacts remains unchanged.CRUD is an option for the implementation of the GOF pattern – adapter, becauseWith it, we adapt the application interfaces to any database, combine incompatible interfaces.
The words are empty, show me the code
To implement abstractions in programming languages, interfaces/protocols/abstract classes are used.All these are phenomena of the same order, however, in interviews you may be asked to name the difference between them, I personally think that there is much sense in this becauseThe only purpose of use is the implementation of data abstraction, otherwise it is an interviewer memory test.
CRUD is often implemented as part of the repository pattern, the repository, however, can implement the CRUD interface, or may not be implemented, it all depends on the ingenuity of the developer.

Consider a rather typical SWIFT code of the Book structures repository, which works directly with the userDefaults database:

import Foundation

struct Book: Codable {
	let title: String
	let author: String
}

class BookRepository {
	func save(book: Book) {
    		let record = try! JSONEncoder().encode(book)
    		UserDefaults.standard.set(record, forKey: book.title)
	}
    
	func get(bookWithTitle title: String) -> Book? {
    		guard let data = UserDefaults.standard.data(forKey: title) else { return nil }
    		let book = try! JSONDecoder().decode(Book.self, from: data)
    		return book
	}
    
	func delete(book: Book) {
    		UserDefaults.standard.removeObject(forKey: book.title)
	}
}

let book = Book(title: "Fear and Loathing in COBOL", author: "Sir Edsger ZX Spectrum")
let repository = BookRepository()
repository.save(book: book)
print(repository.get(bookWithTitle: book.title)!)
repository.delete(book: book)
guard repository.get(bookWithTitle: book.title) == nil else {
	print("Error: can't delete Book from repository!")
	exit(1)
}

The code above seems simple, however, we calculate the number of violations of the principle of Dry (do not repeat youurself) and the connected code:
Continence to the userdefaults database
The connectedness of the json -jsonencoder, jsondecoder encoders and decoders
Correspondence with the Book structure, and we need an abstract repository in order not to create a repository class for each structure that we will store in the database (Dry Violation)

I meet such a CRUD code of the repository quite often, it is possible to use it, but high connectedness, duplication of code, lead to the fact that over time its support will become very complicated.This will be especially felt when trying to switch to another database, or when the internal logic of working with the database in all repository created in the application.
Instead of duplicating the code, keeping high connectedness – we will write a protocol for the CRUD repository, thus abstracting the interface of the database and business logic of the application, observing DRY, carrying out low connectedness:

protocol CRUDRepository {
    typealias Item = Codable
    typealias ItemIdentifier = String
    
    func create(id: CRUDRepository.ItemIdentifier, item: T) async throws
    func read(id: CRUDRepository.ItemIdentifier) async throws -> T
    func update(id: CRUDRepository.ItemIdentifier, item: T) async throws
    func delete(id: CRUDRepository.ItemIdentifier) async throws
}

The Crudrepository protocol describes the interfaces and associated data types for the further implementation of a particular CRUD repository.

Next, write a specific implementation for the UserDefaults database:

class UserDefaultsRepository: CRUDRepository {
    private typealias RecordIdentifier = String
    
    let tableName: String
    let dataTransformer: DataTransformer
    
    init(
   	 tableName: String = "",
   	 dataTransformer: DataTransformer = JSONDataTransformer()
    ) {
   	 self.tableName = tableName
   	 self.dataTransformer = dataTransformer
    }
    
    private func key(id: CRUDRepository.ItemIdentifier) -> RecordIdentifier {
   	 "database_\(tableName)_item_\(id)"
    }
   	 
    private func isExists(id: CRUDRepository.ItemIdentifier) async throws -> Bool {
   	 UserDefaults.standard.data(forKey: key(id: id)) != nil
    }
    
    func create(id: CRUDRepository.ItemIdentifier, item: T) async throws {
   	 let data = try await dataTransformer.encode(item)
   	 UserDefaults.standard.set(data, forKey: key(id: id))
   	 UserDefaults.standard.synchronize()
    }
    
    func read(id: CRUDRepository.ItemIdentifier) async throws -> T {
   	 guard let data = UserDefaults.standard.data(forKey: key(id: id)) else {
   		 throw CRUDRepositoryError.recordNotFound(id: id)
   	 }
   	 let item: T = try await dataTransformer.decode(data: data)
   	 return item
    }
    
    func update(id: CRUDRepository.ItemIdentifier, item: T) async throws {
   	 guard try await isExists(id: id) else {
   		 throw CRUDRepositoryError.recordNotFound(id: id)
   	 }
   	 let data = try await dataTransformer.encode(item)
   	 UserDefaults.standard.set(data, forKey: key(id: id))
   	 UserDefaults.standard.synchronize()
    }
    
    func delete(id: CRUDRepository.ItemIdentifier) async throws {
   	 guard try await isExists(id: id) else {
   		 throw CRUDRepositoryError.recordNotFound(id: id)
   	 }
   	 UserDefaults.standard.removeObject(forKey: key(id: id))
   	 UserDefaults.standard.synchronize()
    }
}

The code looks long, but contains a complete specific implementation of CRUD repository containing weak contact, details further.
Typealias’s are added to make code smug.
Weakness and strong connectivity
The damping from a specific structure (Struct) is implemented using geneticist T, which in turn should implement CODable protocols.Codable allows you to transform structures using classes that implement the Toplevelencoder and ToplevelDecoder protocols, such as JSONENCODER and JSONDECODER, using the basic types (Int, String, Float, etc.), there is no need to write additional code for transforming structures.

The damping from a particular encoder and decoder occurs using abstracting in the Datatransformer protocol:

protocol DataTransformer {
	func encode(_ object: T) async throws -> Data
	func decode(data: Data) async throws -> T
}

Using the implementation of the Date-transformer, we implemented an abstraction of the enkoder and decoder interfaces, having implemented a weak connectedness to ensure work with various types of data formats.

The following is the code of a particular Datatransformer, namely for JSON:

class JSONDataTransformer: DataTransformer {
	func encode(_ object: T) async throws -> Data where T : Encodable {
    		let data = try JSONEncoder().encode(object)
    		return data
	}
    
	func decode(data: Data) async throws -> T where T : Decodable {
    		let item: T = try JSONDecoder().decode(T.self, from: data)
    		return item
	}
}

And so it was possible?

What has changed?Now it is enough to initiate a specific repository for working with any structure that the CODBLE protocol implits, in this way the need for the duplication of code disappears, and a faint connected application is realized.

An example of a client CRUD with a specific repository, the database is userdefaults, the JSON data format, the Client structure, also an example of the recording and reading of the array:

import Foundation

print("One item access example")

do {
	let clientRecordIdentifier = "client"
	let clientOne = Client(name: "Chill Client")
	let repository = UserDefaultsRepository(
    	tableName: "Clients Database",
    	dataTransformer: JSONDataTransformer()
	)
	try await repository.create(id: clientRecordIdentifier, item: clientOne)
	var clientRecord: Client = try await repository.read(id: clientRecordIdentifier)
	print("Client Name: \(clientRecord.name)")
	clientRecord.name = "Busy Client"
	try await repository.update(id: clientRecordIdentifier, item: clientRecord)
	let updatedClient: Client = try await repository.read(id: clientRecordIdentifier)
	print("Updated Client Name: \(updatedClient.name)")
	try await repository.delete(id: clientRecordIdentifier)
	let removedClientRecord: Client = try await repository.read(id: clientRecordIdentifier)
	print(removedClientRecord)
}
catch {
	print(error.localizedDescription)
}

print("Array access example")

let clientArrayRecordIdentifier = "clientArray"
let clientOne = Client(name: "Chill Client")
let repository = UserDefaultsRepository(
	tableName: "Clients Database",
	dataTransformer: JSONDataTransformer()
)
let array = [clientOne]
try await repository.create(id: clientArrayRecordIdentifier, item: array)
let savedArray: [Client] = try await repository.read(id: clientArrayRecordIdentifier)
print(savedArray.first!)

At the first check, CRUD has been processed an exception in which the reading of the remote Aitem will already be unavailable.

Switch the database

Now I will show how to transfer the current code to another database.For example, I will take the code of the SQLite repository that ChatGPT generated:

import SQLite3

class SQLiteRepository: CRUDRepository {
    private typealias RecordIdentifier = String
    
    let tableName: String
    let dataTransformer: DataTransformer
    private var db: OpaquePointer?

    init(
   	 tableName: String,
   	 dataTransformer: DataTransformer = JSONDataTransformer()
    ) {
   	 self.tableName = tableName
   	 self.dataTransformer = dataTransformer
   	 self.db = openDatabase()
   	 createTableIfNeeded()
    }
    
    private func openDatabase() -> OpaquePointer? {
   	 var db: OpaquePointer? = nil
   	 let fileURL = try! FileManager.default
   		 .url(for: .documentDirectory, in: .userDomainMask, appropriateFor: nil, create: false)
   		 .appendingPathComponent("\(tableName).sqlite")
   	 if sqlite3_open(fileURL.path, &db) != SQLITE_OK {
   		 print("error opening database")
   		 return nil
   	 }
   	 return db
    }
    
    private func createTableIfNeeded() {
   	 let createTableString = """
   	 CREATE TABLE IF NOT EXISTS \(tableName) (
   	 id TEXT PRIMARY KEY NOT NULL,
   	 data BLOB NOT NULL
   	 );
   	 """
   	 var createTableStatement: OpaquePointer? = nil
   	 if sqlite3_prepare_v2(db, createTableString, -1, &createTableStatement, nil) == SQLITE_OK {
   		 if sqlite3_step(createTableStatement) == SQLITE_DONE {
       		 print("\(tableName) table created.")
   		 } else {
       		 print("\(tableName) table could not be created.")
   		 }
   	 } else {
   		 print("CREATE TABLE statement could not be prepared.")
   	 }
   	 sqlite3_finalize(createTableStatement)
    }
    
    private func isExists(id: CRUDRepository.ItemIdentifier) async throws -> Bool {
   	 let queryStatementString = "SELECT data FROM \(tableName) WHERE id = ?;"
   	 var queryStatement: OpaquePointer? = nil
   	 if sqlite3_prepare_v2(db, queryStatementString, -1, &queryStatement, nil) == SQLITE_OK {
   		 sqlite3_bind_text(queryStatement, 1, id, -1, nil)
   		 if sqlite3_step(queryStatement) == SQLITE_ROW {
       		 sqlite3_finalize(queryStatement)
       		 return true
   		 } else {
       		 sqlite3_finalize(queryStatement)
       		 return false
   		 }
   	 } else {
   		 print("SELECT statement could not be prepared.")
   		 throw CRUDRepositoryError.databaseError
   	 }
    }
    
    func create(id: CRUDRepository.ItemIdentifier, item: T) async throws {
   	 let insertStatementString = "INSERT INTO \(tableName) (id, data) VALUES (?, ?);"
   	 var insertStatement: OpaquePointer? = nil
   	 if sqlite3_prepare_v2(db, insertStatementString, -1, &insertStatement, nil) == SQLITE_OK {
   		 let data = try await dataTransformer.encode(item)
   		 sqlite3_bind_text(insertStatement, 1, id, -1, nil)
   		 sqlite3_bind_blob(insertStatement, 2, (data as NSData).bytes, Int32(data.count), nil)
   		 if sqlite3_step(insertStatement) == SQLITE_DONE {
       		 print("Successfully inserted row.")
   		 } else {
       		 print("Could not insert row.")
       		 throw CRUDRepositoryError.databaseError
   		 }
   	 } else {
   		 print("INSERT statement could not be prepared.")
   		 throw CRUDRepositoryError.databaseError
   	 }
   	 sqlite3_finalize(insertStatement)
    }
    
    func read(id: CRUDRepository.ItemIdentifier) async throws -> T {
   	 let queryStatementString = "SELECT data FROM \(tableName) WHERE id = ?;"
   	 var queryStatement: OpaquePointer? = nil
   	 var item: T?
   	 if sqlite3_prepare_v2(db, queryStatementString, -1, &queryStatement, nil) == SQLITE_OK {
   		 sqlite3_bind_text(queryStatement, 1, id, -1, nil)
   		 if sqlite3_step(queryStatement) == SQLITE_ROW {
       		 let queryResultCol1 = sqlite3_column_blob(queryStatement, 0)
       		 let queryResultCol1Length = sqlite3_column_bytes(queryStatement, 0)
       		 let data = Data(bytes: queryResultCol1, count: Int(queryResultCol1Length))
       		 item = try await dataTransformer.decode(data: data)
   		 } else {
       		 throw CRUDRepositoryError.recordNotFound(id: id)
   		 }
   	 } else {
   		 print("SELECT statement could not be prepared")
   		 throw CRUDRepositoryError.databaseError
   	 }
   	 sqlite3_finalize(queryStatement)
   	 return item!
    }
    
    func update(id: CRUDRepository.ItemIdentifier, item: T) async throws {
   	 guard try await isExists(id: id) else {
   		 throw CRUDRepositoryError.recordNotFound(id: id)
   	 }
   	 let updateStatementString = "UPDATE \(tableName) SET data = ? WHERE id = ?;"
   	 var updateStatement: OpaquePointer? = nil
   	 if sqlite3_prepare_v2(db, updateStatementString, -1, &updateStatement, nil) == SQLITE_OK {
   		 let data = try await dataTransformer.encode(item)
   		 sqlite3_bind_blob(updateStatement, 1, (data as NSData).bytes, Int32(data.count), nil)
   		 sqlite3_bind_text(updateStatement, 2, id, -1, nil)
   		 if sqlite3_step(updateStatement) == SQLITE_DONE {
       		 print("Successfully updated row.")
   		 } else {
       		 print("Could not update row.")
       		 throw CRUDRepositoryError.databaseError
   		 }
   	 } else {
   		 print("UPDATE statement could not be prepared.")
   		 throw CRUDRepositoryError.databaseError
   	 }
   	 sqlite3_finalize(updateStatement)
    }
    
    func delete(id: CRUDRepository.ItemIdentifier) async throws {
   	 guard try await isExists(id: id) else {
   		 throw CRUDRepositoryError.recordNotFound(id: id)
   	 }
   	 let deleteStatementString = "DELETE FROM \(tableName) WHERE id = ?;"
   	 var deleteStatement: OpaquePointer? = nil
   	 if sqlite3_prepare_v2(db, deleteStatementString, -1, &deleteStatement, nil) == SQLITE_OK {
   		 sqlite3_bind_text(deleteStatement, 1, id, -1, nil)
   		 if sqlite3_step(deleteStatement) == SQLITE_DONE {
       		 print("Successfully deleted row.")
   		 } else {
       		 print("Could not delete row.")
       		 throw CRUDRepositoryError.databaseError
   		 }
   	 } else {
   		 print("DELETE statement could not be prepared.")
   		 throw CRUDRepositoryError.databaseError
   	 }
   	 sqlite3_finalize(deleteStatement)
    }
}

Or CRUD Code for a repository for a file system that also generated ChatGPT:

import Foundation

class FileSystemRepository: CRUDRepository {
	private typealias RecordIdentifier = String
    
	let directoryName: String
	let dataTransformer: DataTransformer
	private let fileManager = FileManager.default
	private var directoryURL: URL
    
	init(
    	directoryName: String = "Database",
    	dataTransformer: DataTransformer = JSONDataTransformer()
	) {
    	self.directoryName = directoryName
    	self.dataTransformer = dataTransformer
   	 
    	let paths = fileManager.urls(for: .documentDirectory, in: .userDomainMask)
    	directoryURL = paths.first!.appendingPathComponent(directoryName)
   	 
    	if !fileManager.fileExists(atPath: directoryURL.path) {
        	try? fileManager.createDirectory(at: directoryURL, withIntermediateDirectories: true, attributes: nil)
    	}
	}
    
	private func fileURL(id: CRUDRepository.ItemIdentifier) -> URL {
    	return directoryURL.appendingPathComponent("item_\(id).json")
	}
    
	private func isExists(id: CRUDRepository.ItemIdentifier) async throws -> Bool {
    	return fileManager.fileExists(atPath: fileURL(id: id).path)
	}
    
	func create(id: CRUDRepository.ItemIdentifier, item: T) async throws {
    	let data = try await dataTransformer.encode(item)
    	let url = fileURL(id: id)
    	try data.write(to: url)
	}
    
	func read(id: CRUDRepository.ItemIdentifier) async throws -> T {
    	let url = fileURL(id: id)
    	guard let data = fileManager.contents(atPath: url.path) else {
        	throw CRUDRepositoryError.recordNotFound(id: id)
    	}
    	let item: T = try await dataTransformer.decode(data: data)
    	return item
	}
    
	func update(id: CRUDRepository.ItemIdentifier, item: T) async throws {
    	guard try await isExists(id: id) else {
        	throw CRUDRepositoryError.recordNotFound(id: id)
    	}
    	let data = try await dataTransformer.encode(item)
    	let url = fileURL(id: id)
    	try data.write(to: url)
	}
    
	func delete(id: CRUDRepository.ItemIdentifier) async throws {
    	guard try await isExists(id: id) else {
        	throw CRUDRepositoryError.recordNotFound(id: id)
    	}
    	let url = fileURL(id: id)
    	try fileManager.removeItem(at: url)
	}
}

We replace the repository in the client code:

import Foundation

print("One item access example")

do {
	let clientRecordIdentifier = "client"
	let clientOne = Client(name: "Chill Client")
	let repository = FileSystemRepository(
    	directoryName: "Clients Database",
    	dataTransformer: JSONDataTransformer()
	)
	try await repository.create(id: clientRecordIdentifier, item: clientOne)
	var clientRecord: Client = try await repository.read(id: clientRecordIdentifier)
	print("Client Name: \(clientRecord.name)")
	clientRecord.name = "Busy Client"
	try await repository.update(id: clientRecordIdentifier, item: clientRecord)
	let updatedClient: Client = try await repository.read(id: clientRecordIdentifier)
	print("Updated Client Name: \(updatedClient.name)")
	try await repository.delete(id: clientRecordIdentifier)
	let removedClientRecord: Client = try await repository.read(id: clientRecordIdentifier)
	print(removedClientRecord)
}
catch {
	print(error.localizedDescription)
}

print("Array access example")

let clientArrayRecordIdentifier = "clientArray"
let clientOne = Client(name: "Chill Client")
let repository = FileSystemRepository(
	directoryName: "Clients Database",
	dataTransformer: JSONDataTransformer()
)
let array = [clientOne]
try await repository.create(id: clientArrayRecordIdentifier, item: array)
let savedArray: [Client] = try await repository.read(id: clientArrayRecordIdentifier)
print(savedArray.first!)

The initialization of userdefaultsrepository is replaced by FilesyStemrepository, with compatible arguments.
After launching the second version of the client code, you will find in the document folder the Clients Database Directory, which will contain a massif serialized in JSON with one Client structure.

Switching data storage format

Now we ask Chatgpt to generate an encoder and a decoder for XML:

class XMLDataTransformer: DataTransformer {
	let formatExtension = "xml"
    
	func encode(_ item: T) async throws -> Data {
    	let encoder = PropertyListEncoder()
    	encoder.outputFormat = .xml
    	return try encoder.encode(item)
	}
    
	func decode(data: Data) async throws -> T {
    	let decoder = PropertyListDecoder()
    	return try decoder.decode(T.self, from: data)
	}
}

Thanks to built -in types in SWIFT, the task for the neural network becomes elementary.

Replace JSON with XML in the client code:

import Foundation

print("One item access example")

do {
	let clientRecordIdentifier = "client"
	let clientOne = Client(name: "Chill Client")
	let repository = FileSystemRepository(
    	directoryName: "Clients Database",
    	dataTransformer: XMLDataTransformer()
	)
	try await repository.create(id: clientRecordIdentifier, item: clientOne)
	var clientRecord: Client = try await repository.read(id: clientRecordIdentifier)
	print("Client Name: \(clientRecord.name)")
	clientRecord.name = "Busy Client"
	try await repository.update(id: clientRecordIdentifier, item: clientRecord)
	let updatedClient: Client = try await repository.read(id: clientRecordIdentifier)
	print("Updated Client Name: \(updatedClient.name)")
	try await repository.delete(id: clientRecordIdentifier)
	let removedClientRecord: Client = try await repository.read(id: clientRecordIdentifier)
	print(removedClientRecord)
}
catch {
	print(error.localizedDescription)
}

print("Array access example")

let clientArrayRecordIdentifier = "clientArray"
let clientOne = Client(name: "Chill Client")
let repository = FileSystemRepository(
	directoryName: "Clients Database",
	dataTransformer: XMLDataTransformer()
)
let array = [clientOne]
try await repository.create(id: clientArrayRecordIdentifier, item: array)
let savedArray: [Client] = try await repository.read(id: clientArrayRecordIdentifier)
print(savedArray.first!)

The client code has changed only one expression jsondatatransformer -> XMldatratransformer

Result

CRUD repository is one of the design patterns that can be used to implement the weakness of the components of the application architecture.Another solution is the use of ORM (object mapping), if briefly, the ORM uses an approach in which the structures are completely tolerate on the database, and then changes with models should be displayed (flashing (!)) On the database.
But this is a completely different story.

The full implementation of CRUD repositories for SWIFT is available on the link:
https://gitlab.com/demensdeum/crud-example
By the way, SWIFT has long been supported outside MacOS, the code from the article was written and tested on Arch Linux.

Sources
https://developer.apple.com/documentation/combine/topleveldecoder
https://developer.apple.com/documentation/combine/toplevelencoder
https://en.wikipedia.org/wiki/Create,_read,_update_and_delete

dd input/output error

If you receive an input/output error when copying a normal disk using dd in Linux, here are the steps you should take:

The situation is quite sad, but solvable. Most likely, you are dealing with a faulty disk that contains bad blocks that can no longer be used, written to, or read from.

Be sure to check such a disk using S.M.A.R.T., it will most likely show you disk errors. This was the case for me, the number of bad blocks was so large that I had to say goodbye to the old hard drive and replace it with a new SSD.

The problem was that this disk had a fully functional system with licensed software necessary for work. I attempted to use partimage to quickly copy the data, but found that the utility only copied a third of the disk, then terminated either with a segfault or some other amusing C/C++ quirk.

Then I tried copying the data using dd, and found that dd reached about the same point as partimage, then encountered an input/output error. Various fun flags like conv=noerr, skip, or other such things did not help at all.

However, I was able to copy the data to another disk without any issues using a GNU utility called ddrescue.

ddrescue /dev/sda1 /dev/sdb1

After that, my hair became silky, my wife returned, the kids, and the dog stopped biting the sofa.

A big advantage of ddrescue is the built-in progress bar, so there is no need to hack together some workarounds like pv and other not very pretty dd flags. Also, ddrescue shows the number of attempts to read the data; the wiki also says that the utility has some super algorithm for reading damaged data, we’ll leave that for people who like to dig into the source code, we are not among them, right?

https://en.wikipedia.org/wiki/Ddrescue
https://www.gnu.org/software/ddrescue/ddrescue.html

ChatGPT Overview

Hi all! In this article, I want to talk about ChatGPT, a powerful language modeling tool from OpenAI that can help solve various text-processing tasks. I will show how this tool works and how it can be used in practical situations. Let’s get started!

At the moment, ChatGPT is one of the best neural network-based language models in the world. It was created to help developers create intelligent systems that can generate natural language and communicate with people in it.

One of the key advantages of ChatGPT is its ability for contextual text modeling. This means that the model takes into account the previous dialogue and uses it to better understand the situation and generate a more natural response.

You can use ChatGPT for a variety of tasks such as customer support automation, chatbot creation, text generation, and more.

The neural networks behind ChatGPT have been trained on huge arrays of text to provide highly accurate predictions. This allows the model to generate natural text that can support dialogue and answer questions.

With ChatGPT, you can create your own chatbots and other intelligent systems that can interact with people in natural language. This can be especially useful in industries such as travel, retail, and customer support.

In conclusion, ChatGPT is a powerful tool for solving various language modeling problems. Its ability for contextual modeling makes it especially useful for building chatbots and intelligent systems.


In fact, everything that ChatGPT wrote above was completely written by itself. What? Yes? I’m shocked myself!

The network itself can be tested here:
https://chat.openai.com/chat

Turn on USB keyboard backlight on macOS

I recently bought a very inexpensive Getorix GK-45X USB keyboard with RGB backlight. After connecting it to a Macbook Pro on an M1 processor, it became clear that the RGB backlight was not working. Even by pressing the magic combination Fn + Scroll Lock, it was not possible to turn on the backlight, only the backlight level of the MacBook screen changed.
There are several solutions to this problem, namely OpenRGB (does not work), HID LED Test (does not work). Only the kvmswitch utility worked:
https://github.com/stoutput/OSX-KVM

You need to download it from the github and allow it to run from the terminal in the Security panel of the System Settings.
As I understood from the description, after launching the utility sends pressing Fn + Scroll Lock, thus turning on/off the backlight on the keyboard.

Tree sort

Tree sort – binary search tree sort. Time complexity – O(n²). In such a tree, each node has numbers less than the node on the left, more than the node on the right, when coming from the root and printing the values ​​from left to right, we get a sorted list of numbers. Surprising huh?

Consider the binary search tree schema:

Derrick Coetzee (public domain)

Try to manually read the numbers starting from the penultimate left node of the lower left corner, for each node on the left – a node – on the right.

It will turn out like this:

  1. The penultimate node at the bottom left is 3.
  2. She has a left branch – 1.
  3. Take this number (1)
  4. Next, take the vertex 3 (1, 3) itself
  5. To the right is branch 6, but it contains branches. Therefore, we read it in the same way.
  6. Left branch of node 6 number 4 (1, 3, 4)
  7. Node 6 itself (1, 3, 4, 6)
  8. Right 7 (1, 3, 4, 6, 7)
  9. Go up to the root node – 8 (1,3, 4 ,6, 7, 8)
  10. Print everything on the right by analogy
  11. Get the final list – 1, 3, 4, 6, 7, 8, 10, 13, 14

To implement the algorithm in code, you need two functions:

  1. Building a binary search tree
  2. Printing the binary search tree in the correct order

They assemble a binary search tree in the same way as they read it, a number is attached to each node on the left or right, depending on whether it is less or more.

Lua example:

Node = {value = nil, lhs = nil, rhs = nil}

function Node:new(value, lhs, rhs)
    output = {}
    setmetatable(output, self)
    self.__index = self  
    output.value = value
    output.lhs = lhs
    output.rhs = rhs
    output.counter = 1
    return output  
end

function Node:Increment()
    self.counter = self.counter + 1
end

function Node:Insert(value)
    if self.lhs ~= nil and self.lhs.value > value then
        self.lhs:Insert(value)
        return
    end

    if self.rhs ~= nil and self.rhs.value < value then
        self.rhs:Insert(value)
        return
    end

    if self.value == value then
        self:Increment()
        return
    elseif self.value > value then
        if self.lhs == nil then
            self.lhs = Node:new(value, nil, nil)
        else
            self.lhs:Insert(value)
        end
        return
    else
        if self.rhs == nil then
            self.rhs = Node:new(value, nil, nil)
        else
            self.rhs:Insert(value)
        end
        return
    end
end

function Node:InOrder(output)
    if self.lhs ~= nil then
       output = self.lhs:InOrder(output)
    end
    output = self:printSelf(output)
    if self.rhs ~= nil then
        output = self.rhs:InOrder(output)
    end
    return output
end

function Node:printSelf(output)
    for i=0,self.counter-1 do
        output = output .. tostring(self.value) .. " "
    end
    return output
end

function PrintArray(numbers)
    output = ""
    for i=0,#numbers do
        output = output .. tostring(numbers[i]) .. " "
    end    
    print(output)
end

function Treesort(numbers)
    rootNode = Node:new(numbers[0], nil, nil)
    for i=1,#numbers do
        rootNode:Insert(numbers[i])
    end
    print(rootNode:InOrder(""))
end


numbersCount = 10
maxNumber = 9

numbers = {}

for i=0,numbersCount-1 do
    numbers[i] = math.random(0, maxNumber)
end

PrintArray(numbers)
Treesort(numbers)

An important nuance is that for numbers that are equal to the vertex, a lot of interesting mechanisms for hooking to the node have been invented, but I just added a counter to the vertex class, when printing, the numbers are returned by the counter.

Links

https://gitlab.com/demensdeum /algorithms/-/tree/master/sortAlgorithms/treesort

References

TreeSort Algorithm Explained and Implemented with Examples in Java | Sorting Algorithms | Geekific – YouTube

Tree sort – YouTube

Convert Sorted Array to Binary Search Tree (LeetCode 108. Algorithm Explained) – YouTube

Sorting algorithms/Tree sort on a linked list – Rosetta Code

Tree Sort – GeeksforGeeks

Tree sort – Wikipedia

How to handle duplicates in Binary Search Tree? – GeeksforGeeks

Tree Sort | GeeksforGeeks – YouTube