Creating Your Own Instrument

Let's say you're a fellow modder who'd like to use this mod as a dependency for making a new instrument. While I heavily doubt anyone would actually do it, I still left a pretty neat API for you to use. In any case, this section is mostly for me to remember my stuff :P

Anywho, this tutorial will assume that you are a complete beginner to the subject of modding and such in general, as it makes it easier for me to write it that way.
Of course, you will still need basic Java, Forge (and Minecraft) knowledge.

Resources

All resources for any instrument's screen must be kept under assets\<modid>\textures\gui\instrument\<instrument>. If this file structure is not followed, the API will not be able to detect your resources.

Our Goal

We're going to start simple, and assume that all you have in mind is a simple grid instrument that inherits the 7x3 (21) notes style - per se, a piano.

_{*Quick sidenote, I'd actually love to implement a piano into this mod but I just don't have the sounds for it. If you actually make it that'd be very epic and I'll be in a life's debt to you.}

NOTE: If you wish to implement an instrument that does not inherit a grid style, you may use the generic AbstractInstrumentScreen. I suggest look into the implementation of Arataki's Great and Glorious Drum.

Creating an Instrument Screen

Firstly, we'll need to create an instrument screen. As the name may suggest, it is the container for our instrument's GUI. Such a screen must extend the AbstractInstrumentScreen class.

NOTE: This is a client-only class; screens do not exist on dedicated servers.

This mod has a convenient implementation for grid-specific instruments called AbstractGridInstrument - so let us use this in our example.
We will therefore annotate it as such:

@OnlyIn(Dist.CLIENT)
public class PianoScreen extends AbstractGridInstrumentScreen {
    public static final String INSTRUMENT_ID = "piano";
    //...
}

The INSTRUMENT_ID field is not required, however we will have to reference the ID more than once, so it is recommended to store it there.
This ID must match that of your item's ID.

This class will kindly ask you to implement some methods. Let's explore their purposes and how you should handle them:

Constructor Method

All that the AbstractInstrumentScreen constructor requires is an InterationHand instance. This hand represents the hand of which the player currently plays the instrument with.

getInstrumentId()

Pretty self-explanatory. Just return the INSTRUMENT_ID we made, paired with your mod ID:

@Override
public ResourceLocation getInstrumentId() {
    return new ResourceLocation(Main.MODID, INSTRUMENT_ID);
}

If you plan on copying an existing source directory, consider skipping texturing.

getThemeLoader()

This method should return an instance of an InstrumentThemeLoader.

This class is responsible for handling the color palettes of the instrument, and will automatically assign them according to the JSON styler associated in the instrument's resources directory, under the name of instrument_style.json. We will explore its contents later.
For now, just know that our lad AbstractInstrumentScreen has got our back, and stores in it a neat method to create one for you. Just call initThemeLoader, passing both your mod and instrument's ID.

Triggering the Theme Loader

The initiator for this class must be called upon game startup. This is so because upon its construction, it automatically subscribes this instrument for any resource load events on the client; and the first one is just about when the game fires.
This means that if we do not call it at that time, your mod's and any resource pack that is modifying your instrument will not have its effects applied until a resource load event occurs (such F3+T being pressed).

The trick I use for it is to make the theme loader a static property of the screen class, and annotate the screen class as an EventBusSubscriber for the mod event bus. This makes the FML load the class early in, quite a lazy way.
We must also declare the value property to be of CLIENT for it to not attempt to load on the server.

Alternatively, you could make a dummy static method in your instrument class and call it upon client setup, or call a forName/loadClass method on said class. Either one will work.

Thus, we end up with the following snippet:

@OnlyIn(Dist.CLIENT)
//NOTE: There just to make it load on mod setup
@EventBusSubscriber(bus = Bus.MOD, value = Dist.CLIENT)
public class PianoScreen extends AbstractGridInstrumentScreen {
    //...

    private static final InstrumentThemeLoader THEME_LOADER = initThemeLoader(PianoMod.MODID, INSTRUMENT_ID);
    @Override
    public InstrumentThemeLoader getThemeLoader() {
        return THEME_LOADER;
    }

    //...

getSourcePath() (optional)

You can define here a different resource location than the one specified earlier to make the API point to it. It is useful for taking existing resources from already existing instruments.

For instance, let us point to the Floral Zither:

@Override
protected ResourceLocation getSourcePath() {
    return new ResourceLocation(GInstrumentMod.MODID, FloralZitherScreen.INSTRUMENT_ID);
}

You will also need to match this entry with your theme loader:

private static final InstrumentThemeLoader THEME_LOADER = initThemeLoader(GInstrumentMod.MODID, FloralZitherScreen.INSTRUMENT_ID);

getSounds() (grid-only)

This method should return an array of NoteSounds. Its length must be equal to the total notes that can be pressed in your instrument, and its order will later be described.

For now, at least, we don't have such array; so let's help ourselves by simply returning null.

noteLayout() (optional)

This method defines the note labels used for this instrument. The default one consists of the regular, and very common layout of: ["C", "D", "E", "F", "G", "A", "B"]. However, if you wish to implement an instrument like the Vintage Lyre, whose notes layout is slightly different, you can do something like the following (source):

public class VintageLyreScreen extends AbstractGridInstrumentScreen {
    //...
    public static final String[] VINTAGE_LYRE_LAYOUT = new String[] {
        "C", "Db", "Eb", "F", "G", "Ab", "Bb",
        "C", "D", "Eb", "F", "G", "A", "Bb",
        "C", "D", "Eb", "F", "G", "A", "Bb"
    };

    //...

    @Override
    public String[] noteLayout() {
        return VINTAGE_LYRE_LAYOUT;
    }
    //...

Note that this array will wrap around once it goes out of bounds, so if you have a layout that spans across all 3 rows, you can just omit the other 2.

isGenshinInstrument()

You most likely would like to return false here. I am the only one who is allowed to set this to true >:)

Nevertheless, it's just an indicator for the mod to pop-up a disclaimer before the first usage of a Genshin instrument. It is likely you don't require this.

isSSTI() (optional)

This boolean will define whether your grid instrument is a Singular Sound-Type Instrument. This means that you only need to provide 1 note sound to the instrument, and it will adjust its pitch automatically for you!

Note that enabling this feature will disable your ability to manually pitch the instrument in-game. It generally is always better to implement the full sounds, though, for the better quality.

To Conclude

With all said and done, we end up with this class at hand:

@OnlyIn(Dist.CLIENT)
//NOTE: There just to make it load on mod setup
@EventBusSubscriber(bus = Bus.MOD, value = Dist.CLIENT)
public class PianoScreen extends AbstractGridInstrumentScreen {
    public static final String INSTRUMENT_ID = "piano";

    public PianoScreen(InteractionHand hand) {
        super(hand);
    }
    @Override
    public ResourceLocation getInstrumentId() {
        return new ResourceLocation(Main.MODID, INSTRUMENT_ID);
    }


    @Override
    public NoteSound[] getSounds() {
        return null;
    }

    @Override
    public boolean isGenshinInstrument() {
        return false;
    }

    private static final InstrumentThemeLoader THEME_LOADER = initThemeLoader(Main.MODID, INSTRUMENT_ID);
    @Override
    public InstrumentThemeLoader getThemeLoader() {
        return THEME_LOADER;
    }
    
}

NOTE: You can also modify the amount of rows/columns in a grid instrument. Simply override the rows()/columns() methods. Do note that the default keybinds will match up to 8 rows and 3 columns. If you wish to extend or modify this in any way, override noteMap as well, providing the mappings of keys to note buttons.
For a grid, all you must do is simply pass in to noteGrid.genKeyboardMap your key mappings of choice in the form of a 2D array, and return its value.

• ALL notes must be put in the notes map.

Adding Sounds

NoteSound

Mono and Stereo

Because of some annoying specification by OpenAL, the sound library used by Minecraft - all sounds produced on Stereo will be stuck on 100%.
I have thus made a system to balance quality and your bloody ears, by implementing the NoteSound class. There, you may define a mono sound and an optional stereo version. Which one is chosen depends on multiple factors, but ultimately boils down to the user's preference, as set by their instrument's options;

If the player has chosen either Mono or Stereo sounds only, all instrument sounds are forced into the respected version.
However, when Mixed (the default options) is chosen:

• If the instruments volume is lesser than 100%, mono sounds will always play. This does not account for the master volume.
• When requested by the client, stereo sounds will always by chosen.
• When requested by the server, only if the distance between the player and the client is lesser than the set stereo range (5 blocks), stereo will be chosen.

Note that if a stereo version is not supplied, then quite obviously - mono will always be chosen.

Embedding Sounds

For SSTI Instruments

All you have to do is just declare which SoundEvent you wish to have your instrument tune for you. Something like so:

public static final NoteSound[] BANJO = new NoteSound[] {
    new NoteSound(NoteBlockInstrument.BANJO.getSoundEvent().get(), Optional.empty())
}

To implement your own custom SoundEvents, see this Forge tutorial.

You may now skip to the end.

Location & Convention

All instrument sounds should be kept under: assets/<modid>/sounds/<instrument>/<note_number>.ogg. If you have their stereo version too, name them <note_number>.stereo.ogg by convention.

NOTE: The sounds, in a grid, must be ordered from the first visual row to the last, as described here:

In our case, we would need 21 piano key sounds. Let us assume that we also happen to have a stereo version of the piano as well - we will thus include all 42 audio files in our instruments folder, as described earlier.

NOTE: If you only have the stereo versions of your sounds, you can channel them down to a singular mono channel:

1. Using Audacity: Just find a quick YouTube tutorial on the matter. This one is nice.

2. Using FFMPEG: This option is most useful for a large batch of files.
   The following command was generated by ChatGPT, and it worked quite well for me. Make sure to run it in your instrument > sounds directory as the active directory.

   PowerShell:

   Get-ChildItem -Filter "*.ogg" | ForEach-Object { ffmpeg -i $_.FullName -ac 1 -c:v copy ($_.BaseName + "_mono" + $_.Extension) }

   Unix Bash (Untested!):

   for file in *.ogg; do ffmpeg -i "$file" -ac 1 -c:v copy "${file%.*}_mono.${file##*.}"; done

   In summary, the command takes all the ".ogg" files in the current directory and converts each file's audio to mono using ffmpeg. The converted files will have "_mono" appended to their filenames before the original file extension.
   It is still recommended to rename your mono output files to *.ogg and the stereo ones to *.stereo.ogg in order to follow the mod's conventions, though.

sounds.json

Forge requires you make a JSON file describing all your sounds that the mod adds, as described here.
The keys of these sounds will be what Minecraft and the GI-API will use to reference your sounds. The naming of the sounds must follow this naming principal: <instrument>_note_<note number>, and for stereo types: <instrument>_note_<note number>_stereo.
All instrument sounds should channel to the records category.

So, your file should be constructed as something like this:

{
    "piano_note_0": {
        "catrgory": "record",
        "sounds": ["pianomod:piano/0"]
    },
    "piano_note_1": {
        "category": "record",
        "sounds": ["pianomod:piano/1"]
    },
    //...
    "piano_note_20": {
        "category": "record",
        "sounds": ["pianomod:piano/20"]
    },

    // If you have stereo versions:
    "piano_note_0_stereo": {
        "catrgory": "record",
        "sounds": ["pianomod:piano/0.stereo"]
    },
    "piano_note_1_stereo": {
        "category": "record",
        "sounds": ["pianomod:piano/1.stereo"]
    },
    //...
    "piano_note_20_stereo": {
        "category": "record",
        "sounds": ["pianomod:piano/20.stereo"]
    }
}

Do notice how the sounds array refers to the location inside of our sounds directory: piano's the parent directory and the files range from 0 to 20.

Registering the sounds to the API

Registering the sounds is made extremely easy on your end. And I really mean it - all you need is but a single method to handle it all.
You will need the general registration class, of type SoundEvent, and its contents will be as followed:

public abstract class ModSounds {
    
    public static final DeferredRegister<SoundEvent> SOUNDS = DeferredRegister.create(ForgeRegistries.SOUND_EVENTS, PianoMod.MODID);
    public static void register(final IEventBus bus) {
        SOUNDS.register(bus);
    }

    public static final NoteSound[] PIANO_NOTE_SOUNDS = NoteSoundRegistrer.createInstrumentNotes(
        SOUNDS, new ResourceLocation(PianoMod.MODID, "piano"), true
    )

}

createInstrumentNotes will simply register to the specified registrer all the sounds necessary for the instrument to function - those that you put in the sound directory previously, as well as constructing an according NoteSound object for it.

NOTE: The NoteSound objects created in this manner will not contain their sounds until Forge's registration events!

The true argument at the end defines whether this instrument has stereo support. You can either replace it with false or remove this argument to specify otherwise.
Additionally, if your instrument is not a 3x7 grid, you may specify the rows and columns for your instruments after the stereo boolean, such as:

NoteSoundRegistrer.createInstrumentNotes(SOUNDS, new ResourceLocation(PianoMod.MODID, "piano"), true, 5, 6)

Where the grid here would be 5x6.

NOTE: You may also define singular note sounds and not a grid of them. An example can be found on the glorious drum's implementation:

GLORIOUS_DRUM = new NoteSound[] {
    NoteSoundRegistrer.registerNote(SOUNDS, loc("glorious_drum_don")),
    NoteSoundRegistrer.registerNote(SOUNDS, loc("glorious_drum_ka"))
}

NOTE: loc() is just a shorthand for the resource location within my mod ID.

You may append a boolean argument for this method, too, to define stereo support for it.

NoteSounds not part of a grid instrument must define their own NoteButton to handle their sounds.

Wrapping it up

Let's get back to our PianoScreen. now, All we now need to do is instead of returning null in our getSounds() method - we will return the array of sounds we've just made:

@Override
public NoteSound[] getSounds() {
    return ModSounds.PIANO_NOTE_SOUNDS;
}

Texturing

Instrument Styler

instrument_style.json is a JSON styler file that should be present under your instrument's resources directory. It (currently only) defines the color scheme of our instrument. 3 of its properties are customizable:

{
    "note_theme": [30, 30, 30],
    "label_theme": [20, 20, 20],
    "note_pressed_theme": [255, 249, 239]
}

• note_theme defines the RGB color of the note text when the note is unpressed, as well as the ring produced when pressing the notes. It should be about the same color of the note button's pressed state. I'm thinking of black-ish piano keys, so I'll put all values on 30.
• label_theme defines the RGB color of the note labels (these symbols on the notes) when the note is unpressed. It should be the same color of the note button's pressed state, so I'll put the values on 20.
• note_theme_pressed defines the RGB color of the note text when the note is pressed. It should be the same as the main color of the note button's unpressed note state. By default, this value is set to 255, 249, 239, so if you did not change the aforementioned note color - you can simply omit this property.

NOTE: If you want to add any other design elements to the styler, you may subscribe to a theme load event via InstrumentThemeLoader#addListener in a subclass, replacing it in your instrument's screen:

public class PianoThemeLoader extends InstrumentThemeLoader {

   private Color nailsColor;
   private int parentDissapointmentAmount;
   //...

   public PianoThemeLoader(ResourceLocation instrumentStyleLocation) {
       super(instrumentStyleLocation);
       addListener(this::loadPianoStuff);
   }

   private void loadPianoStuff(JsonObject themeLoader) {
       setNailsColor(getTheme(themeLoader.get("nails_color"), Color.BLACK));

       final JsonElement dissapointment = themeLoader.get("dissapointment");
       if (dissapointment.isJsonPrimitive())
           setDissapointmentAmount(tryGetProperty(dissapointment.getAsJsonPrimitive(), JsonPermitive::getAsInt, 1));

       //...
   }

   // Getters & setters etc.

}

It is recommended that you make use of InstrumentOptionsScreen for most options, though - simply override initInstrumentOptionsScreen.
Only use the styler for properties that make sense for a resource pack to modify.

Note Background

The note background refers to the note texture by itself. You must place the texture in your instrument's resources folder under the note directory, named "note_bg.png". An example of such texture may be found here:

It is divided into 3 parts:

1. The unpressed note state
2. The pressed note state
3. Hovered/focused/foreign player state

Note Labels

By default, the note grid provides you with a label sprite of 7 items representing the do-re-mi/ABC. It is located under genshinstrument:textures/gui/instrument/grid_notes.png. You may change this to your liking by overwriting the getNoteSymbolsLocation() method in your instrument's screen. The colors taken for the textures are taken from the instrument styler, as described earlier.

NOTE: The grid instrument will shift the displayed label based on its offset to the ABC. If you want to make it be consistent to their set rows (or any other variable as such), you may override the updateNoteLabel() method under your NoteGridButton() (override initNoteGrid() to customize the buttons on your grid):

@Override
public void updateNoteLabel() {
    super.updateNoteLabel();
    noteRenderer.noteTextureRow = row; // or anything else
}

Texture Model

The Genshin Instruments mod provides you with an instrument model to inherit: genshinstrument:item/instrument. This is a base model that inherits the regular minecraft:item/generated base model, however it also applies the transformations that make the instrument appear correctly in the player's hands.
Here is an example for a piano.json item model:

{
    "parent": "genshinstrument:item/instrument",
    "textures": {
        "layer0": "pianomod:item/piano"
    }
}

Sending an Open Packet

To open our instrument, through any means, we need to send a packet from the server to the client. This is done so that the server could register the player opening it, as well as to provide a way of the client classes to not leak to the server.

The mod uses OpenInstrumentPacket; It will essentially take any instrument's ID as an argument, and call Minecraft's setScreen() method with a new instance of your screen via a map. Feel free to use this implementation or another that suits your needs.

Making an Instrument Item

An instrument item is an extension of the InstrumentItem class. There is no need to make a subclass for it - you may simply initiate it directly in your registrater class, like so:

public static final RegistryObject<Item> PIANO = ITEMS.register("piano", () ->
    new InstrumentItem(
        (player, hand) -> ModPacketHandler.sendToClient(new OpenInstrumentPacket("piano", hand), player)
    )
)

NOTE: You can also add item properties in a 2nd argument, however the item stack will always be set to 1.

The method you must provide to the constructor is responsible for sending an open instrument packet to the given player. DO NOT USE ANY OF THE PROVIDED PACKETS / PACKET HANDLER. Make sure to implement your own.

Making an Instrument Block

A piano is most likely a block rather than an item.

An instrument block is any block that derives from an AbstractInstrumentBlock:

public class PianoInstrumentBlock extends AbstractInstrumentBlock {

    public PianoInstrumentBlock(Properties pProperties) {
        super(pProperties);
    }

    @Override
    public InstrumentBlockEntity newBlockEntity(BlockPos pPos, BlockState pState) {
        return new CustomInstrumentBlockEntity(pPos, pState);
    }

    @Override
    protected OpenInstrumentPacketSender instrumentPacketSender() {
        return (player, hand) -> ModPacketHandler.sendToClient(new OpenInstrumentPacket("piano", hand), player)
    }
    
}

Notice that you will need to create your own CustomInstrumentBlockEntity for this to work. See BlockEntities.

the instrumentPacketSender method is the same as the item's argument. Use your own packet handler and open packet.

NOTE: Modifying the used arm pose for playing can be achieved by overriding the initClientBlockUseAnim method.

Wrapping Up

By this point, your instrument should already be fully integrated into the game. Heydad! 🙌
This is a very "in a nutshell" tutorial - you can extend many more methods, and apply much more functionality of your own using the provided API. I encourage you to further explore its limitations, and produce whatever is to your liking.

I will also, as a side-note, mention that ServerUtil provides you with 2 methods to manually send sound packets of 2 types: by player and non-player. With it, you can make any event, block, or idk what else to produce instrument sounds - and not just the regular instruments. As per se, this mod is quite extensible; so just go nuts.