It's hard to avoid programming overcomplicated monoliths if none
of your programs can talk to each other.
Unix tradition strongly encourages writing programs that
read and write simple, textual, stream-oriented, device-independent
formats. Under classic Unix, as many programs as possible are written
as simple filters, which take a simple text
stream on input and process it into another simple text stream on
output.
Despite popular mythology, this practice is favored not because
Unix programmers hate graphical user interfaces. It's because if you
don't write programs that accept and emit simple text streams, it's
much more difficult to hook the programs together.
Text streams are to Unix tools as messages are to objects in an
object-oriented setting. The simplicity of the text-stream interface
enforces the encapsulation of the tools. More elaborate forms of
inter-process communication, such as remote procedure calls, show a
tendency to involve programs with each others' internals too
much.
To make programs composable, make them independent. A program
on one end of a text stream should care as little as possible about
the program on the other end. It should be made easy to replace one
end with a completely different implementation without disturbing the
other.
GUIs can be a very good thing. Complex binary data formats are
sometimes unavoidable by any reasonable means. But before writing a
GUI, it's wise to ask if the tricky interactive parts of your
program can be segregated into one piece and the workhorse
algorithms into another, with a simple command stream or
application protocol connecting the two. Before devising a tricky
binary format to pass data around, it's worth experimenting to see
if you can make a simple textual format work and accept a little
parsing overhead in return for being able to hack the data stream
with general-purpose tools.
When a serialized, protocol-like interface is not natural for the
application, proper Unix design is to at least organize as many of the
application primitives as possible into a library with a well-defined
API. This opens up the possibility that the application can be called
by linkage, or that multiple interfaces can be glued on it for differenttasks.