It all depends in how you manage it. If you pick up manure regularly in combination with dragging or harrowing, and mow aggressively during the growing season, yes, it's doable. You would be better off cross fencing the area so you could rotate and rest the grazing. You'd be better off still if you had a small area where you could lock all the animals off the grazing when it was especially wet, or if they were in a stall or shed for part of the day.
This question gets asked often, and in a lot of different ways, but the answer is always "It depends." Depends on the initial condition of the pasture, how you manage and maintain it and a host of other factors. My general rule of thumb is the smaller the area, the more labor intensive and more expensive it is.
I'm not sure I would want to do what you're proposing, I think given the cost of supplementing hay and grain, you'd end up losing money on the calf, as you'll be paying for every pound you put on him. Raising a calf makes sense if you can put the weight on cheaply. And your scheme is a little labor intensive for my taste, you're going to spend a lot of time picking manure, dragging and feeding. Just my opinion, your mileage may vary.