[go: nahoru, domu]

Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Forwarding into Beta [1.0.0-BETA] #8

Merged
merged 113 commits into from
Sep 28, 2023
Merged

Forwarding into Beta [1.0.0-BETA] #8

merged 113 commits into from
Sep 28, 2023

Conversation

ShindouMihou
Copy link
Owner
@ShindouMihou ShindouMihou commented Aug 27, 2022

As part of compatibility changes for v3.6.0 of Javacord, Nexus is introducing a step forward into beta release, including a magnitude of breaking changes and improvements that will help shape your bot and allow for further scaling.

Note

These changes are not final yet and there might be more to be added, but this serves as a baseline for the future. (this note will be removed after this pull request is merged).

Warning

These changes are written for Javacord v3.6.0-SNAPSHOT (and non-snapshot), if you are planning on using this, please use Javacord v3.6.0 otherwise this will not work.

🧭 Changes

This pull request may not contain the full magnitude of changes but will include as many of the breaking changes as possible and provide a way to resolve the differences.

📟 Synchronizer

The Nexus Synchronizer has been rewritten from the bottom up once more to be better and simpler. It's major change has to do with how the new synchronization methods are written and also with its behavior where it will now use the return values from Discord to create an in-memory index map (contrary to the prior version which did nothing and discarded the result).

All public APIs in regard to the synchronizer have not been changed but the synchronize methods have undergone quite some change with the interface being rewritten to look like the following:

public interface NexusSynchronizeMethods {
    CompletableFuture<Set<ApplicationCommand>> bulkOverwriteGlobal(DiscordApi shard, Set<SlashCommandBuilder> slashCommands);
    CompletableFuture<Set<ApplicationCommand>> bulkOverwriteServer(DiscordApi shard, Set<SlashCommandBuilder> slashCommands, long serverId);
    CompletableFuture<Void> deleteForServer(DiscordApi shard, NexusCommand command, long serverId);
    CompletableFuture<ApplicationCommand> updateForServer(DiscordApi shard, NexusCommand command, long serverId);
    CompletableFuture<ApplicationCommand> createForServer(DiscordApi shard, NexusCommand command, long serverId);
}

The previous version looked a lot more like this:

public interface NexusSynchronizeMethods {
    CompletableFuture<Void> bulkOverwriteGlobal(DiscordApi shard, List<SlashCommandBuilder> slashCommands);
    void bulkOverwriteServer(DiscordApi shard, List<SlashCommandBuilder> slashCommands, long serverId, CompletableFuture<Void> future);
    void deleteForServer(DiscordApi shard, NexusCommand command, long serverId, CompletableFuture<Void> future);
    void updateForServer(DiscordApi shard, NexusCommand command, long serverId, CompletableFuture<Void> future);
    void createForServer(DiscordApi shard, NexusCommand command, long serverId, CompletableFuture<Void> future);
}

The changes in how the synchronize methods are created reflect the new behavior of the synchronizer which now utilizes the return values. You can check the default synchronize methods to have a guide over how to write the methods.

Another particular breaking change with the synchronizer involves how you change the synchronize methods, previously it used to be like this:

NexusSynchronizer.SYNCHRONIZE_METHODS.set(new YourOwnSynchronizeMethods());

In the beta version, it has been simplified to look like this:

NexusSynchronizer.SYNCHRONIZE_METHODS = new YourOwnSynchronizeMethods();

🚒 EngineX

For all the people who have been using EngineX, these new changes will definitely be a major improvement, especially for people who are using sharding. Nexus EngineX (coincidentally named how NGINX is called) now has better support for CompletableFutures with the following new methods introduced:

CompletableFuture<DiscordApi> await(int shard);
CompletableFuture<Server> await(long server);
CompletableFuture<DiscordApi> awaitAvailable();
void failFutureOnExpire(NexusEngineEvent event, CompletableFuture<U> future);

All await methods use the queue feature of EngineX internally while also using the new failFutureOnExpire which will listen to expiration changes to fail the CompletableFuture if no shard has taken the request after the expiration time has been reached (default: 15 minutes).

An example of how this is being used is:

Nexus nexus = ...
new DiscordApiBuilder()
       .setToken(...)
       .addListener(nexus)
       .setTotalShards(4)
       .loginAll()
       .forEach(future -> future.thenAccept(shard -> {
            nexus.getShardManager().put(shard);
            //... other stuff like onShardLogin()
       }).exceptionally(ExceptionLogger.get()));

nexus.getEngineX().awaitAvailable().thenAccept(shard -> System.out.println("Shard" + shard.getCurrentShard() + " is processing this one");

It's a simple example that demonstrates how to create Nexus and how to use EngineX at the same time. EngineX is more of a shard router that routes "events" onto shards to process them without creating an entire messy logic to wait for the shards themselves.

EngineX listens to this line (nexus.getShardManager().put(shard)) to know when a shard is available.

The example given simply prints out "Shard X is processing this one" to console with X being the shard that announces their presence first to Nexus.

Another breaking change with EngineX's existing APIs has to do with the store that had no real purpose. I don't really remember what the store was added for but for all the bots that I've written with Nexus, the store has been unused and the internals also doesn't use it. Therefore, the store has been removed. The new signature of EngineX events looks like this:

(shard) -> ...

In an example method, it would look like this:

nexus.getEngineX().queue((shard) -> System.out.println("Shard " + shard.getCurrentShard() + " is processing this one"));

Additionally, the EngineX should now be more safer against thread issues (e.g. deadlocks or race conditions).

📦 Command Manager

The command manager has been extended to include support methods for people who want to export their in-memory index map that was created by Nexus into their data stores (e.g. Redis or any database), it will also support importing the stored indexes to save time processing.

In particular, the following methods were added:

/**
* Exports the indexes that was created which can then be used to create a database copy of the given indexes.
* <br><br>
* It is not recommended to use this for any other purposes other than creating a database copy because this creates
* more garbage for the garbage collector.
*
* @return A snapshot of the in-memory indexes that the command manager has.
*/
List<NexusCommandIndex> export();


/**
* Creates a new fresh in-memory index map by using the {@link NexusCommandManager#index()} method before creating
* an export of the given indexes with the {@link NexusCommandManager#export()} command.
*
* @return A snapshot of the in-memory indexes that the command manager has.
*/
List<NexusCommandIndex> indexThenExport();

/**
* This indexes all the commands whether it'd be global or server commands to increase
* performance and precision of slash commands.
*/
void index();

/**
* Creates an in-memory index mapping of the given command and the given slash command snowflake.
* <br><br>
* You can use this method to index commands from your database.
*
* @param command   The command that will be associated with the given snowflake.
* @param snowflake The snowflake that will be associated with the given command.
*/
void index(NexusCommand command, long snowflake);

/**
* Massively creates an in-memory index mapping for all the given command using the indexer reducer
* provided below.
* <br><br>
* This is a completely synchronous operation and could cause major blocking on the application if you use
* it daringly.
*
* @param indexer The indexer reducer to collectively find the index of the commands.
*/
void index(Function<NexusCommand, Long> indexer);

/**
* Creates an in-memory index of all the slash commands provided. This will map all the commands based on properties
* that matches e.g. the name (since a command can only have one global and one server command that has the same name)
* and the server property if available.
*
* @param applicationCommandList the command list to use for indexing.
*/
void index(Set<ApplicationCommand> applicationCommandList);

/**
* Creates an in-memory index of the given slash command provided. This will map the command based on the property
* that matches e.g. the name (since a command can only have one global and one server command that has the same name)
* and the server property if available.
*
* @param command The command to index.
*/
void index(ApplicationCommand command);

To export the in-memory index, you can use the following:

Nexus nexus = ...
List<NexusCommandIndex> indexes = nexus.getCommandManager().export();

The NexusCommandIndex is composed of two properties:

  • command() which returns the NexusCommand itself.
  • applicationCommandId() which returns the stored application command identifier from Discord.

You can then store those two properties in any data store that you like e.g. Redis then import them with the indexer method or the other methods (if you prefer, the synchronizer should already index all commands though):

Nexus nexus = ...
nexus.getCommandManager().index((command) -> {
     return Long.parseLong(Redis.getItem("command:" + command.getName()));
});

Alternatively without the indexer method:

Nexus nexus = ...
nexus.getCommandManager().getCommands().forEach(command -> {
    nexus.getCommandManager().index(command,  Long.parseLong(Redis.getItem("command:" + command.getName()));
});

Other than those changes, the command manager has been changed internally to be more verbose and is now being used by the synchronizer actively.

🌏 Core Changes

The Nexus Core itself has been changed minorly but might be breaking for some. In particular, all properties related to Nexus.start() like the onShardLogin and DiscordApiBuilder from the Nexus Builder have been removed as there is no real purpose for using those methods.

Nexus shouldn't be responsible for creating your shards or booting them up as this leaves little control for the developers to create their own clustering system which is important for larger bots.

If you followed the examples on the README.md and other related examples which never used the following methods then you should be fine ⚡.

📄 Installation

You can install this version of Nexus ahead of time by using Jitpack: 1.0.0-beta-SNAPSHOT

…sage).

This adds baseline new features and improvements for the beta release of Nexus which includes many breaking changes to introduce much better functionality.

As part of this change, the entire NexusSynchronizer and its default methods have been modified to create indexes on-the-go on anything that effects command additions.

Nexus' EngineX has new methods such as `queue(predicate, event)`, `await(shard)`, `await(server)`, `awaitAvailable()` and `failFutureOnExpire(event, future)` which are explained deeper in the pull request (#8).

All references to a writeable record store in Nexus EngineX have been removed because it has no clear purpose.

Nexus Shard Manager now has a conveience method for the shard id calculation `((server >> 22) % totalShards)` which is called `shardOf(server, totalShards)`

Nexus Command Manager can now support importing indexes from the database and exporting its current in-memory indexes, more details on PR (#8).

`Nexus.start()` has been removed and methods related to creating shards and login with shards in `NexusBuilder` has been removed.
…rtFor(serverIds)` in NexusCommand.

This change is correlated with making Nexus a bit more clearer to understand. It's more preferred to use the word disassociate and associate when it comes to servers.
@ShindouMihou
Copy link
Owner Author

Additional breaking changes were included to have better support for multiple shards, especially in multi-cluster environments.

🧷 Nexus Synchronizer

A new breaking change was introduced to the synchronizer to better optimize the synchronizer for multi-cluster environments wherein the server shard may not be in the current cluster. The following changes have been applied:

- CompletableFuture<Void> synchronize(int totalShards)
+ CompletableFuture<Void> synchronize()

- CompletableFuture<Void> batchUpdate(long serverId, DiscordApi shard)
+ CompletableFuture<Void> batchUpdate(long serverId)

- CompletableFuture<Void> batchUpdate(long serverId, int totalShards)

In summary, the changes are as follows:

  • synchronize no longer requires the totalShards value.
  • batchUpdate no longer requires the DiscordApi value
  • a convenience method for batchUpdate was removed because there was no need for it anymore.

@ShindouMihou
Copy link
Owner Author

Additional changes were implemented onto the command events themselves to make life easier for the developers.

💬 Command Events

More ways to send a response over to Discord has been added for commands which should help sending responses so much easier. The following methods have been added:

fun respondNowWith(contents: String): CompletableFuture<InteractionOriginalResponseUpdater>
fun respondNowWith(vararg embeds: EmbedBuilder): CompletableFuture<InteractionOriginalResponseUpdater>

fun respondNowEphemerallyWith(contents: String): CompletableFuture<InteractionOriginalResponseUpdater>
fun respondNowEphemerallyWith(vararg embeds: EmbedBuilder): CompletableFuture<InteractionOriginalResponseUpdater>

And the following methods were removed since there is no real use for the following:

- fun getOptions(): List<SlashCommandInteractionOption>
- fun getSubcommandOptions(name: String): List<SlashCommandInteractionOption>

You can do now the following:

override fun onEvent(event: NexusCommandEvent) {
    event.respondNowWith("Pong!").exceptionally(ExceptionLogger.get())
}

It looks more verbose and cleaner than the following:

override fun onEvent(event: NexusCommandEvent) {
    event.respondNow().setContent("Pong!").respond().exceptionally(ExceptionLogger.get())
}

@ShindouMihou
Copy link
Owner Author

As Javacord v3.6.0 is now released, this PR needs to be fast-forwarded a bit more. To outline the final changes we need:

  • Kotlin-first
  • Support any new changes from Javacord, if any.
  • Adds metedata indexing to help with bots that need to get the full information about the indexes such as which server this command belongs to, etc. This will replace the original indexing module by adding a two-parameter object with two fields (server?, command).
  • and many others to follow.

An optimistic deadline for this change will be by November when I will be more free.

@ShindouMihou
Copy link
Owner Author

Nexus has officially transitioned into a Kotlin-first framework with majority of its functionality written in Kotlin. You can read the full changes of the Kotlin-first here.

An additional change that was implemented is Index Stores which is how we operate the Nexus Command Manager's Indexes from hereon as a move to flexibility with the framework. You can now customize the Index Store to collect indexes from the database, etc.

View IndexStore interface
interface IndexStore {

    /**
     * Adds the [NexusMetaIndex] into the store which can be retrieved later on by methods such as
     * [get] when needed.
     * @param metaIndex the index to add into the index store.
     */
    fun add(metaIndex: NexusMetaIndex)

    /**
     * Gets the [NexusMetaIndex] from the store or an in-memory cache by the application command identifier.
     * @param applicationCommandId the application command identifier from Discord's side.
     * @return the [NexusMetaIndex] that was caught otherwise none.
     */
    operator fun get(applicationCommandId: Long): NexusMetaIndex?

    /**
     * Gets the [NexusMetaIndex] from the store or an in-memory cache by the Nexus unique identifier.
     * @param command the unique identifier of the command, tends to be of the same value always unless the name changed or
     * the unique identifier was changed via the [IdentifiableAs] annotation.
     * @return the [NexusMetaIndex] that was caught otherwise none.
     */
    operator fun get(command: String): NexusMetaIndex?

    /**
     * Adds one or more [NexusMetaIndex] into the store, this is used in scenarios such as mass-synchronization which
     * offers more than one indexes at the same time.
     *
     * @param metaIndexes the indexes to add into the store.
     */
    fun addAll(metaIndexes: List<NexusMetaIndex>)

    /**
     * Gets all the [NexusMetaIndex] available in the store, this is used more when the command manager's indexes are
     * exported somewhere.
     *
     * @return all the [NexusMetaIndex] known in the store.
     */
    fun all(): List<NexusMetaIndex>

    /**
     * Clears all the known indexes in the database. This happens when the command manager performs a re-indexing which
     * happens when the developer themselves has called for it.
     */
    fun clear()

}

A meta index is simply a data class that contains the following properties:

data class NexusMetaIndex(val command: String, val applicationCommandId: Long, val server: Long?)

Properties:

  • command: the Nexus command's unique identifier
  • applicationCommandId: the slash command's identifier in Discord.
  • server: the server where the command is assigned. (this should be highly accurate if the indexes came from synchronization).

To understand how persistent indexes work, we have to understand a new key component in indexing and that is the unique identifier of a command. In Discord, there can be commands with the same name (one global, one per server) and that's fine until it is designed in a framework.

Nexus has been designed to assign the command name as the unique identifier of the command, but this has a problem, when two commands have the same name but different scopes, both have an index identifier conflict. It may seem like an issue at first, but there is a simpler fix to this and that is assigning a different unique identifier for the command.

The framework has been designed to identify potential index-identifier conflicts at runtime (when you add commands at Nexus), if Nexus finds that a command has the same unique identifier then it will throw an IndexIdentifierConflictException which tells you the instructions on how to resolve it.

To resolve the issue, all you need to do is add the @IdentifiableAs("...") annotation which overrides the command's unique identifier into another value. For example, let us say that we have two conflicting commands (one global, one server):

class PingCommand {
    val name = "ping"
}

class PingCommandServer {
   val name = "ping"
}

Nexus will complain that the two commands have the same unique identifier which leads to a IndexIdentifierConflictException but this can be easily resolved by changing one of the commands' unique identifier such as it becomes:

class PingCommand {
    val name = "ping"
}

@IdentifiableAs("ping-server")
class PingCommandServer {
   val name = "ping"
   ...
}

And that resolves the conflict, the two commands will operate as usual. (in terms of hierarchy, server commands take more priority than global commands).

…hods and uses JVM annotations to make JVM use better, removes unused Pair class and resolves a possible memory leak.
@ShindouMihou
Copy link
Owner Author

Nexus 1.0 is now ready for official release with the following latest changes and breaking changes from the 1.0.0-next.

Breaking Changes

NexusCommandInterceptor

Methods to add interceptors such as middlewares and afterwares in the NexusCommandInterceptor interface has been removed completely in favor of Nexus.interceptors, this is to keep the API more consistent across the board.

Migration:

- NexusCommandInterceptor.addMiddleware("key") { ... }
+ Nexus.interceptors.middleware("key") { ... }
-  NexusCommandInterceptor.middleware { ... }
+ Nexus.interceptors.middleware { ... }

NexusCommandInterceptorRepository

We're removing this altogether because there is no point in bringing this when it simply just does the same thing as the newer style but under a define function.

Migration:

- object SomeRepository: NexusCommandInterceptorRepository {
-    val SOME_MIDDLEWARE: String = "some.middleware"
-    override fun define() {
-        middleware(SOME_MIDDLEWARE) { ... }
-    }
- }
+ object SomeRepository {
+    val SOME_MIDDLEWARE: String = Nexus.interceptors.middleware("some.middleware") { ... }
+ }

cooldown in NexusCommand

We've removed cooldown field as a native field in NexusCommand, this is, in order to keep things more consistent, as we do not offer the Ratelimiter as a Nexus-Native middleware (like OptionValidation is).

Migration:

- val cooldown = Duration.ofSeconds(5)
+ @Share val cooldown = Duration.ofSeconds(5)

NexusCommonInterceptors

We recently changed NexusCommonInterceptors from a interface to an singleton, so this may cause some import issues for some people. Don't worry, just reimport it!

NexusLoggingAdapter

We recently changed the internal methods of Nexus to pass exceptions in the parameters of Nexus.logger without any placeholders (not that we even use placeholders now as we use Kotlin), so you may see exceptions being passed as a parameter even though there are no placeholders.

NexusCommand

We've removed older methods that have long been marked for deprecation, the following functions were moved:

- fun addSupportFor(vararg serverIds: Long): NexusCommand
- fun removeSupportFor(vararg serverIds: Long): NexusCommand
- fun getServerId(): Long

Migration:

val command = ...
command.associate(serverIds)
command.disassociate(serverIds)

# Same behavior as the old `NexusCommand#getServerId`
command.serverIds.first()

Major Changes

Context Menus

First-class support for context menus (user and message) in Nexus is now supported. This will make it so that we don't have to use some sketchy methods to prevent your context menus from being overridden, and this makes it easier to use Nexus with context menus.

object TestUserContextMenu: NexusUserContextMenu() {
    val name = "test"
    override fun onEvent(event: NexusContextMenuEvent<UserContextMenuCommandEvent, UserContextMenuInteraction>) {
        event.respondNowEphemerallyWith("Hello")
    }
}

object TestMessageContextMenu: NexusMessageContextMenu() {
    val name = "test"
    override fun onEvent(event: NexusContextMenuEvent<MessageContextMenuCommandEvent, MessageContextMenuInteraction>) {
        event.respondNowEphemerallyWith("Hello")
    }
}
Nexus.contextMenus(TestUserContextMenu, TestMessageContextMenu)

Failed Dispatch Afterwares

Afterwares can now listen to failed dispatch events which will significantly help with logging and analytical afterwares. Previously, afterwares weren't able to execute after a command failed to dispatch (e.g. a middleware rejecting dispatch), but now, we can listen to these events and even know which middleware caused the problem by using NexusCommandEvent.get(NexusAfterware.BLOCKING_MIDDLEWARE_KEY).

A prime example of this mechanism is the new NexusCommonInterceptors.LOG middleware:
image

object  SomeAfterware: NexusAfterware {
   override fun onAfterCommandExecution(event: NexusCommandEvent) {}
   override fun onFailedDispatch(event: NexusCommandEvent) {} // optional
}

log common afterware

We now have our first common afterware, and it's the NexusCommonInterceptors.LOG afterware which helps you log commands easily. This uses the Nexus.logger to log.

image

Minor Changes

  1. Some behavioral changes have been done to NexusConsoleLoggingAdapter.
    • We now log to stderr instead of stdout for errors.
    • Exceptions are now handled through the logging adapter.
  2. Added more utilities to NexusMessage (Kotlin-only).
    • EmbedBuilder.toNexusMessage()
    • String.toNexusMessage()
  3. Added support for inheritance parents with superclass fields, this will make it so parent inheritance can extend upon superclasses (e.g. abstract classes) and still copy the fields from there.
  4. Added support for superclasses in commands (e.g extending abstract classes).
  5. Nexus will now automatically use NexusConsoleLoggingAdapter when there is no SLF4J logger (as the default logger caused issues wherein exceptions would be ignored as there was no adapter to handle them a.k.a NOP)
  6. A new reflection engine has replaced the previous, making things significantantly easier to add and also making things easier to maintain.
  7. Added some incomplete support for nsfw field for both context and slash commands.
    • Due to Javacord issues, we cannot update the application command's nsfw field using the SlashCommandUpdater because it's missing on the current stable version of Javacord. This does not affect Nexus.synchronizer.synchronize() and Nexus.synchronizer.batchUpdate(server) as those methods uses bulkOverwrite which overwrites the entire commands altogether, allowing updates for those.
  8. You can now include additional ApplicationCommandBuilder to Nexus.synchronizer through Nexus.synchronize.include(server, commands). This is intended when you want to add your own little other stuff e.g. custom context menus that doesn't use Nexus.
  9. Added NexusCommandEvent.get(key) that returns an Object (or Any for Kotlin) as there are cases where our little type-casting doesn't really work, so we'd prefer doing it on our own.
  10. Fixed wiki links on documentations.
  11. A warning is now sent when you try to add @Inherits to an inheritance parent (as that isn't really supported), @Inherits should only be on children.
  12. We now use Nexus.launcher to dispatch all events, so you can now use coroutines or related.

@ShindouMihou ShindouMihou marked this pull request as ready for review September 28, 2023 14:26
@ShindouMihou ShindouMihou merged commit 86ebbd0 into master Sep 28, 2023
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant