A genre of Etude or Study (etude is a French term for a study) is a very particular type of music pieces created in order to develop a performers playing/singing technique (in the case of singers, it is more frequently called Vocalise). A definition from Merriam-Webster dictionary states it as follows: " 1. a piece of music for the practice of a point of technique, 2. a composition built on a technical motive but played for its artistic value". Since I am a pianist, I will talk about the piano etudes.
A long time ago, it became clear to many composers and pianists that certain sort of exercises need to be created in order to help the learners in their formative stages of their piano and music learning. Sometimes such a sort of exercises formed actual pieces of real artistic music, but written predominantly for didactic purposes and the students were able to learn two important lessons from each of those pieces: the elements of music theory and the elements of piano (harpsichord/clavichord) technique. In this type of "etudes" which are simultaneously both instrumental as well as musically invaluable lessons, belong many "didactic" pieces by J. S. Bach, Domenico Scarlatti and some others. To this end, J. S. Bach was writing his famous Little Preludes, Two Part Inventions and Three Part inventions (Sinfonias) and even the great "Well Tempered Harpsichord", books 1 and 2. In addition to these, he compiled a little book of menuets and other little forms called "The Anna Magdalena Notebook". Scarlatti composed his no less than 555 Sonatas, wonderful short sonatas in binary form, full of technical challenges and musical values, which he simply called "Exercises". For both the composers, the main objectives of creating these pieces were actually didactic purposes for both the technical and the musical development, but their musical and artistic values elevate them and make them very fine concert pieces and pieces to enjoy oneself while practising in intimate surroundings. As such, they belong to the second definition from the aforementioned dictionary quotation. In line with this type of studies, to which belong the great, famous and highly artistic etudes by Chopin, Liszt, Rachmaninoff and Debussy, we have quite another type of etudes. In that group of studies, the didactic side of the matter is much more evident because the technical objectives are much emphasised and of the prime importance, whilst the artistic values are given in small doses and therefore are in the background. Many hundreds of studies of this sort were written throughout musical history and there is an ample supply of this sort of materials to choose from for many piano learners. Studies written by Duvernoy, Lemoine, Bertini, Heller, Czerny, Moscheles, Clementi, Cramer and some others provide excellent working and didactic etude literature, which are indispensable in forming a well trained and rounded pianist, even in the case of "recreational" pianists, those piano fellows playing just for pleasure. Working on one's piano technique is a very important part of piano studies because it enables the learner (and any piano lover taking lessons just for pleasure, inevitably is a learner), to cope with pieces of real actual music in a more successful way and opens up many repertoire options due to enriched and better trained mechanical/technical capabilities of the pianist. The technical and theoretical elements found in repertoire pieces are numerous and much varied. The task of didactic sort of etudes is to single out a particular technical element and to deal with it in a variety of ways training the learner and making him/her more competent in that particular technical element, but at the same time it is forming a musical piece in its own right. So, it doesn't sound just as a dry technical exercise (which are also important, but much less fun to have due to a more obvious drill). Some of etudes are more artistic than others and could almost be performed in concerts, but some others are more school-like working materials. However, even in that case, they are musical pieces, meaning that they are formally rounded musical compositions which can still be enjoyed as music. The reason why inexperienced pianists cannot go to study some repertoire pieces straight away, without the use of etudes, scales and arpeggios, is that they lack the theoretical and technical/mechanical knowledges and skills and instant coping with real music and all the requirements which are found in there, would be a very lengthy and complicated process, perhaps even resulting with a sense of worthlessness and consequently a self discouragement.
The luxury of avoiding scales, arpeggios and formal didactic etudes can only be enjoyed by professional pianists who are on such a level that the need of using any of the didactic materials mentioned above is long ceased. For the professionals, all the technical training is in the pieces of music they are working on, but they know all too well what all of the elements found in there represent, what they mean and how to accomplish successfully the musical pieces they work on. However, all that skill and knowledge was learned and trained previously in their lives and now they enjoy the fruits of all the effort input earlier in their lives. Even at that stage, there might be occasional need for some of didactic materials, depending on some particular momentary circumstances. That is one of the reasons why studies by Chopin, Liszt, Rachmaninoff, Debussy and some others exist and serve a double purpose of the technical drill as well as being wonderful and exciting concert pieces, much loved worldwide.
I would strongly encourage all piano learners of whatever stage and goals, to work on etudes suitable for their level, no matter if they are piano learners for pleasure or for graded exams, and to enhance their piano studies and piano and musical enjoyment.