In theory - any language that is turing complete. In Practice you are going to use assembly and c/c++
C is closer to the hardware and compiles instructions that are performant enough for the requirements demanded of an os. To properly boot a system you will probably have to arrange some assembly. Java is designed to generate bytecode not specialized machine code.
This is a very complex area of Computer Science. That being said, it can be undertaken, and, of course, has been undertaken.
As @fight_dragons has stated, C is closer to the hardware and, I believe, this is what early UNIX-based systems were coded in.
To learn C, I’d recommend you take a look at this book:
This is recognised as the gold-standard for the language, despite it being over 40 years old.
In addition to this, I’ve linked an academic article from a university local to me that expresses how to write a simple operating system. Just be aware that this is an academic article and will most likely presume prior knowledge on certain subjects:
Hey, I am also building an OS right now as my project which I am going to submit in college. If you don’t mind, can we work together?
I am using this (also posted by digital0457018858) book and also referring to this YouTube series
Well to be 100% honest, I don’t think I could be much help to yet. I’m still learning the ins and outs of just the C language and have barely glanced at languages like Assembly. I thank you for the offer, but I don’t think I could assist you on that low level of software yet.